Evaluating Safety and Crime Prevention Policies in Europe [1 ed.] 9789054875888

Citation preview

Evaluation_Safety.book Page 1 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

Evaluation_Safety.book Page 2 Tuesday, May 26, 2009 12:46 PM

Evaluation_Safety.book Page 3 Tuesday, May 26, 2009 12:46 PM

Philippe Robert (ed.)

Evaluating Safety and Crime Prevention Policies in Europe Contributions from Tim Hope, Philippe Robert, Sybille Smeets, Carrol Tange, Karin Wittebrood, Anne Wyvekens, Renée Zauberman

This edited book is the outcome of a seminar held within a Coordination Action Assessing Deviance, Crime and Prevention in Europe (CRIMPREV, www.crimprev.eu) funded by the European Commission (contract 028300 under FP6) and coordinated by the Groupe européen de recherches sur les normativités (GERN, Centre national de la recherche scientifique, CNRS).

Evaluation_Safety.book Page 4 Tuesday, May 26, 2009 12:46 PM

Criminological Studies In this series studies are published which contribute significantly to the development of the criminological sciences, through either theoretical elaborations, empirical research or criminal policy discussions. Studies originating from other scientific disciplines can also be considered if they are relevant to the criminological domain. The editorial board consists of academics from the criminological research institutes of the Vrije Universiteit Brussel (VUB) and the Universiteit Gent (UG). All manuscripts are reviewed by at least two referees. Editorial Board: Prof. Dr. J. Christiaens (VUB and UG), president Prof. Dr. T. Decorte (UG) Prof. Dr. C. Eliaerts (VUB) Prof. Dr. S. Gutwirth (VUB) Prof. Dr. P. Hebberecht (UG) Prof. Dr. S. Snacken (VUB and UG)

Cover design: Koloriet, Leefdaal Book design: Style, Hulshout Print: Flin Graphic Group, Oostkamp © 2009 VUBPRESS Brussels University Press VUBPRESS is an imprint of ASP nv (Academic and Scientific Publishers nv) Ravensteingalerij 28 B-1000 Brussels Tel. ++ 32 (0) 2 289 26 50 Fax ++ 32 (0) 2 289 26 59 E-mail [email protected] www.vubpress.be ISBN 978 90 5487 588 8 NUR 824 Legal Deposit D/2009/11.161/047 All rights reserved. No parts of this book may be reproduced or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher.

Evaluation_Safety.book Page 5 Tuesday, May 26, 2009 12:46 PM

Contents

1. Introduction Philippe Robert, Renée Zauberman

7

2. Evaluation of Safety and Crime Prevention Policies in Europe Philippe Robert

11

3. Evaluation of Safety and Crime Prevention Policies in Belgium Sybille Smeets, Carrol Tange

41

4. Evaluation of Safety and Crime Prevention Policies in France A Problematic Activity Anne Wyvekens

61

5. Evaluation of Safety and Crime Prevention Policies in the Netherlands Karin Wittebrood

77

6. Evaluation of Safety and Crime Prevention Policies in England and Wales Tim Hope

91

7. On the Authors

123

Evaluation_Safety.book Page 6 Tuesday, May 26, 2009 12:46 PM

Evaluation_Safety.book Page 7 Tuesday, May 26, 2009 12:46 PM

Introduction Philippe Robert, Renée Zauberman

Stated in very general terms, evaluating a policy means estimating its impact. While the evaluation of safety and crime prevention policies is currently a widely debated issue, both in official discourse and academic circles, a variety of practices come under this heading, ranging from mere incantation to highly sophisticated techniques. Moreover, the apparent favour in which evaluation is held more often than not hides genuine reluctance before actual carrying out. Finally, conditions required for a real evaluation are the subject of serious controversy. This topic has been addressed in a Coordination Action called Assessing Deviance, Crime and Prevention in Europe (CRIMPREV) and funded by the European Commission, within FP6. More specifically, Safety and Crime Prevention Policies Evaluation is one of the topics addressed by a workpackage of this Coordination Action, devoted to Methodology and Good Practices.

I.

The Methodology and Good Practices Workpackage

The aim of this Workpackage was to deliver a synthetic review of the most significant knowledge tools in the field of crime and of their use This entailed: 앫 the mapping of the European situation, 앫 the recognition of good as well as bad practices, 앫 opening the way to European comparisons. During the last half-century, new and powerful tools emerged for the knowledge of crime. They have in common to free themselves from the institutional data in which the study of crime had traditionally closeted itself. Not only have these instruments renewed, even if only partially, the academic knowledge of crime, but they have a high potential as decisionmaking aids. Yet, they have been initiated under varying circumstances across Europe and their mastering is unequally distributed, the number of confirmed specialists being quite small. Consequently, their use is not always

7

Evaluation_Safety.book Page 8 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

appropriate and users outside the academic sphere have often only limited knowledge of their potential. This makes room within CRIMPREV for a Workpackage devoted to a review of the most significant operations of these instruments in the main European countries, in the hope of establishing and circulating both a state of knowledge and a catalogue of good practices in the field. The workpackage is under the joint responsibility of an academic centre, the French Centre de recherches sociologiques sur le droit et les institutions pénales (CESDIP), a professor at the Universitat central de Barcelona, a European network of local communities sharing urban safety concern and programmes, the European Forum for urban safety (EFUS) and an Italian regional organisation coordinating local safety programmes (Città Sicure). This makes this workpackage an interface between the academic and public policies spheres. Four methods have been selected: 앫 general population surveys on victimisation and insecurity, 앫 general population surveys on self-reported crime and deviance, 앫 comparisons between survey-based data and others based on institutional sources, like police statistics, 앫 evaluative research on crime prevention and safety policies. For each of these methods, we have proceeded according to a six-phase protocol: 앫 phase 1: elaboration of a report grid; selection of a general rapporteur in charge of synthesising the information collected, and rapporteurs – a half-dozen per topic – in charge of drawing up the state of knowledge and usage in different countries where the method is sufficiently developed. Of course, given problems of availability of the approached experts, we could not cover them all. We were not aiming at exhaustiveness but rather at presenting a reasonably representative selection of what is being done in the European zone, especially in the principal countries; 앫 phase 2: drafting by each rapporteur of a review pertaining to the country or zone of which she/he is in charge and circulation of these drafts among all rapporteurs; 앫 phase 3: meeting of all rapporteurs, national and general, with the promoters of the workshop in a seminar where the reports are presented and discussed; 앫 phase 4: drafting by the general rapporteur of a synthesis of the reports and discussions; 앫 phase 5: validation of this document by the workpackage promoters and dissemination in the form of a 50-page booklet in English and French, published by the FESU; 8

Evaluation_Safety.book Page 9 Tuesday, May 26, 2009 12:46 PM

Introduction

앫 phase 6: publication of all the contribution in two volumes, one in English, the second in French, both edited by a member of the steering group of the workpackage. Two first volumes have already been published in this series in 2008 and 2009, based on the first workshops of the workpackage: Zauberman R., (Ed.), Victimisation and Insecurity in Europe ; a review of Surveys and their use, Zauberman R., (Ed.), Self-Reported Crime and Deviance Studies in Europe; Current State of Knowledge and Review of Use.

II.

The workshop on Evaluation of Safety and Crime Prevention Policies

While, for the other workshops – surveys on victimisation and insecurity, surveys on self-reported delinquency, comparison between surveys and official statistics – finding rapporteurs was quite an easy task, at least in those European countries where these practices were sufficiently widespread, we encountered many difficulties in forming a team to take up the evaluation issue1. It was therefore decided not to try and cover a large number of European countries but to constitute a spectrum of countries where the expression ‘evaluation of crime prevention and safety policies’ refers to very divergent practices. We were still not at the end of our troubles, for the German rapporteurs, who had agreed to participate, dropped out a few days before the seminar, after which there was no sign of them. Ultimately, the team formed for this workshop comprised: 앫 Sybille Smeets and Carrol Tange (U.L. Bruxelles), for Belgium, 앫 Tim Hope (Keele U.), for England and Wales, 앫 Anne Wyvekens (Centre d’études et de recherches de sciences administratives et politiques – CERSA – CNRS and U. Panthéon-Assas) for France, 앫 Karin Wittebrood (Sociaal en Cultureel Planbureau, SCP) for the Netherlands, 앫 For the general report, Philippe Robert (Centre de recherches sociologiques sur le droit et les institutions pénales – CESDIP – CNRS, U. Versailles Saint Quentin and Department of Justice), in place of Wolfgang Heinz (U. Konstanz) who was originally supposed to take up the assignment,

1.

Thus the various Scandinavian rapporteurs who were solicited, eventually did not take up our collaboration offer.

9

Evaluation_Safety.book Page 10 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

앫 Renée Zauberman (Centre de recherches sociologiques sur le droit et les institutions pénales – CESDIP), Amadeu Recasens i Brunet (Universitat central de Barcelona), Gian Guido Nobili (Città sicure), Michel Marcus (European Forum for urban Safety – EFUS) formed the core group. In October 2007, the national rapporteurs received a letter commissioning the work from the steering committee of the workpackage and submitted their report in June 2008. After this a seminar, chaired by Philippe Robert, was held at the University of Bologna and was attended by the core group, the general rapporteur and the national rapporteurs during a three-day session in July 2008, in order to present and discuss the reports, and draw cross-national lessons. This constituted the basis for the final report, which was first published as a separate English-French booklet (Robert, 2009a). The present publication comprises it together with the national reports Beside this English publication, this book in will also be published in French (Robert, 2009b) in the subseries Déviance et Société at L'Harmattan (Paris). Both publications have been edited by Philippe Robert ; Renée Zauberman has revised the translations.

References ROBERT PH., 2009a, Evaluation of Safety and Crime Prevention Policies in Europe; L’évaluation des politiques de sécurité et de prévention en Europe, Paris, Forum Européen pour la Sécurité Urbaine. ROBERT PH., Dir., 2009b, L’évaluation des politiques de sécurité et de prévention en Europe, Paris, L'Harmattan.

10

Evaluation_Safety.book Page 11 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe1 Philippe Robert

Evaluation constitutes a paradoxical subject: everyone extols its virtues but in reality they are all rather wary. On the policy-makers’ side – political and administrative – it would be delightful to have proof (with an aura of science) that it works, but they are always apprehensive, as Tim Hope (this volume) said, that the policy on which their success and reputation are based, may not prove to be as good as claimed. On the academics’ side, the difficulty of the method can prove a problem. There is always the dread of the ridicule, i.e. having pronounced a programme to be effective, only to later discover that it is counterproductive. Most of all, they dread having their arm twisted by the sponsors who will only accept a laudatory evaluation. Thus Sybille Smeets and Carrol Tange (this volume) show that Belgian academics agreed to participate in the evaluation bids because it was a means of penetrating areas until then closed to investigation rather than because of any great liking for evaluative research. All in all, each protagonist, policy-maker or scholar, has perhaps a lot to gain potentially through evaluation, but also runs the risk of losing enormously. And perhaps this is what explains the real reluctance to work on a theme which however everyone talks about. In the first section, we will first expose the advantages and disadvantages of evaluation. We will then show the variety of ways it has been used in countries selected as typical examples. Then we will explain how we can benefit from an ostensible evaluation without assuming the risks. It is only afterwards that we will examine the evaluation criteria, its prerequisites and know-how.

1.

Translation Priya Sen, revised by the Author and Renée Zauberman

11

Evaluation_Safety.book Page 12 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

I.

The advantages and disadvantages of evaluation2

1.

The reasons that argue in favour of evaluation

Strengthening the policies that are initiated with evaluative mechanisms can be an effective means of pre-empting criticism or having a ready answer for it. A policy-maker who does not bother to have his enterprise evaluated may end up with nothing to say and no counter arguments to make the day he has to face criticism. A programme whose impact is not measured will always attract a Martinson (1974) declaring that nothing works. And even when the criticism is without a very sound basis, or is of mediocre quality, indeed in bad faith, its effects can be devastating if, as a preemptive strategy, regular impact studies have not been carried out. Policies that could have marshalled good arguments in their favour were thus discontinued simply because their promoters had not provided the means for proving their usefulness in time. Hence, evaluation can first of all help to counter ill-founded criticism; but all disputes are not devoid of reason. When a policy or a programme has not undergone any evaluation for a long time, it could have gone off track so drastically without anyone noticing that, by the time anyone realised this, it would have already become so hugely detrimental and useless, and since such a long time, that it would be altogether too late to right the helm: in fact it would seem advisable to abandon it all together or change course radically3. Sherman et al. (1997) give a good example with their history of the Law Enforcement Administration Agency. As a result the impression will be left that a lot of energy and money has been expended for nothing. To go in for evaluations can help prevent such a fatal outcome. Over and above the defensive functions, measuring the impact of a project can help to modify and adjust the plan of action. Safety and crime prevention policies cover such a vast field that their implementation is inevitably unstable and that the helm has to be constantly righted. Indeed these policies encompass all conceivable measures to curb criminal behaviour. The deterrent qualities attributed to law enforcement are far from using them up, very far. In fact, only one chapter in Lawrence Sherman’s report (et al., 19974) to the American Congress on crime prevention5 is devoted to them; the others concern neighbourhood, family, school, work, criminogenic sites, and lastly the police whose actions are 2. 3. 4. 5.

See e.g. Robert, 2003, 2009. Example in Robert, 1994. Extended reprint: Sherman et al., 2002, 2006. A report that reviews some five hundred evaluations spread over 40 years.

12

Evaluation_Safety.book Page 13 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

not only repressive, but preventive as well. We can draw up a very long list: the whole range of social relations is a storehouse to draw upon. Each of these options has its weaknesses. The very general ones run the risk of missing the target, of melting away without any notable preventive effect: this is the reproach made in the 1960s to the great supersaturating urban programmes6. They also run the risk of being in contradiction with the law enforcement policies conceived without any concern for synergy, as conciliating the social and the penal facets of crime control is an uneasy task. Thus the reproach often made to policing aimed at illegal immigration and banned substances is that it can mess up efforts to prevent the ghettoization of disadvantaged urban zones by creating a permanent climate of confrontation in these areas between the residents and the institutions. As for special preventive actions, it can happen that they lack the necessary backing/support structure in a very hostile environment. Thus Sherman (et al., 1977) remarks that preventive programmes cannot single-handedly combat the destructive effects of urban and social segregation in urban hyper-ghettos. In such a situation, special interventions can result in exacerbated safety concerns that can have pernicious effects. In view of these dangers, safety policies rarely subscribe to one model – or at least rarely for any length of time. More generally, or at least sooner or later, they combine the models in variable proportions. Crime prevention constantly oscillates between the too general and the too specific. It is by nature an unstable public policy whose trajectory needs constant adjustments. Accompanying an initiative with an evaluation constitutes a useful tool for the policy-maker who wants to avoid the tyranny of fashionable trends and the pressure of public opinion or the selfinterested advocacy of the most influential professionals. The evaluative discourse can especially help to make the action accountable, not only vis-à-vis the political authorities who decide and finance, but also vis-à-vis the people affected by it and others living in the vicinity. This use is often neglected, perhaps because it requires an effort: it obviously needs a suitable language in order to be understood. It can however be highly useful in preventing antagonisms and encouraging participation. Often local authorities complain about the difficulty of involving the local residents; it should be stressed that often they do not avail of the means at hand. There should be at least one serious discussion on the basis of a reliable report on the positive and negative impact of the programmes or the expected outcome from the projects in the light of previous evaluations. This use of evaluation can also shed light on the safety expectations: the authorities obviously tend to think that their action 6.

See e.g. the report in Robert, Lascoumes, 1974.

13

Evaluation_Safety.book Page 14 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

naturally fits social expectations. It gives them the opportunity to verify if the effect of their action fulfils these expectations. To tell the truth, the problem is a more complex one. In the field of public safety as in other areas, expectations are generally diverse and often contradictory: more often than not the expectations of youth groups from neglected neighbourhoods cannot be the same as those of downtown shopkeepers or pensioners living in residential suburbs. The art of politics consists of combining divergent expectations to as great an extent as possible7, and making trade-offs between incompatibles, based on the specific social compromise on which, in a given society, the legitimacy of public action rests. An account of the impact of programmes in terms that can be understood by diverse audiences will make it easier to explain the choices, which form the basis of the policies, and also to highlight possible pernicious effects, which would otherwise remain unperceived.

2.

Resisting evaluation

Why then, with all its merits, does evaluation meet with such strong opposition? The – little avowed – reluctance of officials, national and local, to what for them is after all a control should be taken into account. It is not that all complain about the obligation to be accountable, but if they admit to the necessity, it is often solely for the benefit of political debate, with which they are familiar. Less readily do they accept an approach, which they consider technocratic. In some countries, evaluation is even suspected of being a managerial concept that denies the specificity of public action, preferring instead modes of reasoning and action appropriate for private industry. There is some truth in this objection however, although managerial primacy favours monitoring or auditing, which chiefly verify whether the implementation of the programme corresponds to the stated objectives and calculating its cost effectiveness8, rendering it a mere caricature of evaluative research. Sometimes the reluctance of the authorities is due to a less noble concern than that of protecting the sphere of political debate from managerial 7.

8.

Anne Wyvekens (2008) gives interesting examples of prevention schemes aimed at teaching people with heterogeneous expectations to co-exist in public places in Paris, such as the Parc de la Villette or the main railway stations. Examples in Leigh (2000) – regarding the obligation of a 2% improvement of efficiency per annum in the Crime and Disorder Act – and Braga et al. (1999), who stress the importance of Value for Money estimation,, in the reform programme of the New Jersey City police.

14

Evaluation_Safety.book Page 15 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

interference. For example, keeping some leeway may be a concern: thus, in the general crime prevention schemes, local authorities find in funding programmes accommodating ways for promoting social action without specifically targeting crime prevention. Or it may simply be the reluctance to account for their actions, the conviction that the policy maker knows what is useful and good for society, thus rendering all verification superfluous. The evaluation can also be missing, not because of the reluctance of the authorities for one reason or another, but because their expectations are too divergent to leave the evaluators sufficient margin: torn between the strong assumptions of the national political forces, the contradictory demands of the Department of Finance, the police, social workers, teachers, doctors, prosecutors, judges, local elected representatives, associations… they are not able to gain sufficient autonomy for serious work. Ultimately, nothing is evaluated and the trade-offs are made only on the basis of the ongoing power struggle between the different protagonists. Some measures resist evaluation more than others. First, strictly speaking, an entire policy can only be evaluated by a sort of meta-evaluation, which is the sum of the evaluations of the different programmes that compose it. After the Dutch Court of Audit had criticised the cost-effectiveness of the country's crime prevention policies, L. Van Noije and Karin Wittebrood (2008) had to review 149 evaluations to evaluate the policy Naar een veiliger samenleving (Towards a Safer Society), launched in 2002 by the first Balkenende government. In addition, it has often been maintained that social prevention measures are more difficult to evaluate than situational crime prevention measures. Thus Karin Wittebrood (this volume) states that programmes that claim to tackle the supposed causes of deviance such as missing school, the quality of parenting, alcoholism, are unevaluable in so far as this causal relation remains hypothetical9. The difficulty probably arises from the fact that social policies only indirectly endeavour to reduce crime and insecurity, i.e. they try to contribute to such effects through their direct impact on the living conditions of the target populations. Lastly, surprisingly, law enforcement is much less subject to evaluation than prevention10, although it has greater need as it depends overwhelmingly on 9.

For all that, Hope (this volume) recalls that the New Deal for Communities Programme (NDC) which pursues reduction of unemployment, of crime, improvement in the fields of health and employment, regeneration of housing and environment in 39 particularly underprivileged localities, was the subject of a national evaluation programme. 10. One of the merits of the Dutch meta-evaluation described by Wittebrood (this volume) consists precisely of not skipping the penal sphere. See however Mackenzie, 2006.

15

Evaluation_Safety.book Page 16 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

unproved assertions11 and the evaluation faces fewer difficulties for it entails individual and mandatory participation of the target-population whereas prevention seeks to mobilise participation which is collective and voluntary12. Faced with such a contradictory situation we will first begin by reviewing the evaluation methods adopted for prevention and safety policies in some European countries.

II.

A variety of evaluative approaches

Rather than cover a large number of European countries, we have constituted a sort of sample of the very varied practices and even conceptions of the evaluation of safety and crime prevention policies. In Belgium the evaluative effort – which is linked to safety contracts between the Federal State and the municipalities – seems to have lost momentum in the last few years. France offers an example of an ‘official’ evaluation generally undertaken by the administration. The Netherlands has chosen a meta-evaluation that examines all the available country-wide data on the basis of ‘quasi-experimental’ scientific canons of Anglo-Saxon origin. Lastly, England and Wales present old and systematic evaluative techniques which cover the entire range of prevention policies, making it possible to address complex methodological problems.

1.

A fast shrinking evaluation13

The safety contracts signed between the Federal Ministry of the Interior and the municipalities were accompanied by evaluations to decide the continuation, extension or reduction of the funds released by the Central government. The totality of these ‘evaluations’ are called ‘summative’ in the sense that their aim is to make an assessment of the contracts between the State and a municipality in order to decide whether to continue it or not. Some evaluations – termed ‘administrative’ – are in the form of tables giving an overview of the number of contracts and the localities involved, 11. As Lawrence Sherman said in 1997 about the massive recourse to imprisonment. 12. As Tim Hope points out (this volume). 13. Based on the report on the Belgium example presented by Sybille Smeets and Carrol Tange (this volume and ref. cit.) at the Bologna seminar.

16

Evaluation_Safety.book Page 17 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

budgetary allocations, the mechanisms retained, and sometimes police statistics or data taken from the Security Monitor14. Others – termed internal – have to be carried out by agents paid by the municipality benefiting from a security contract. But, since 2005, these evaluations have become (preliminary) safety audits – mandatory since 2007 – carried out on the basis of a methodological guide developed by the Ministry of the Interior. Along with this, a research programme was initiated in 1985, again by the Ministry of the Interior, firstly on the police. From the commencement of the following decade, it was extended to safety and crime prevention policies in order to determine the weaknesses in the programmes and to contribute to their improvement. This was called ‘formative’ evaluation. During the entire last decade of the 20th century, academic research teams thus studied the procedures and functioning of the programmes on the local police, the community policing, rapprochement between the police and the citizens, assistance to victims, re-entry into the labour force, hooliganism, school dropouts, and project coordination… This involved mainly qualitative research, which did not specifically plan to measure the impact on crime and insecurity. From 1999, the calls for tender responded to the need of greater operationality. They were addressed to government bodies and private structures. The amount contracted for carrying out the research and its duration was considerably reduced. Far from seeking to evaluate the impact of the programmes and the policies, these studies are more like an inventory of resources, or suggested methods of management, of policy planning and of data analyses. Nevertheless, the results of these works are not really heeded by their sponsors, except if their angle is to ‘know thyself in order to best justify thyself’. Until then, public policies tried to compromise between a conservative trend more in favour of reinforcing the punitive measures, and a socialdemocrat trend, which opted for local safety contracts. Similarly, the evaluations made a sort of compromise between an assessment of the outcome of the programme and a qualitative study of the discourse and personal experience of the actors. In recent years, the emphasis has been on streamlining the resources devoted to safety and the concentration of evaluation funds on straightforward management studies in which the academic sphere is increasingly sidelined. 14. A questionnaire prepared by the police and administered over the phone by a private firm to a random sample of some 40,000 interviewees. The questions were on neighbourhood problems, victimisation, insecurity, contacts with the police and opinions about its functioning. It has been administered 6 times since 1997.

17

Evaluation_Safety.book Page 18 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

2.

An administrative evaluation15

Since the Rocard government of the late 1980s, the evaluation of public policies has become a totally official activity. A statutory instrument was even promulgated some ten years later: Decree 98-1048 of November, 18, 1998 on the evaluation of public policies. This activity seems to be generally conducted by the concerned departments, mostly with the support of the inspectorate which exists in each administrative branch. The most prestigious among them, the Inspection Genérale des Finances (IGF), holds a pre-eminent place, generally in association with the inspection belonging to the relevant ministry. One of the permanent tasks of the Cour des Comptes, the French court of audit, is also the evaluation of public policies. The place of these two bodies underlines the highly ‘financial’ bent of the official conception of evaluation ‘à la française’: primarily it involves verifying the proper use of public monies. With regard to safety and crime prevention policies, however, we can only cite a limited number of examples. In 1993, the conseiller d'État JeanMichel Belorgey had presided the evaluation mission of the politique de la Ville16 but without giving a very prominent place to crime prevention programmes which had been clubbed with this public policies package from the late 1980s. This report was a follow up to the review of Neighbourhood Social Development local contracts – the ancestor to the politique de la Ville – which was completed in 1988 under the direction of François Lévy, an ingénieur général des Ponts et Chaussées (Engineer General of Bridges and Highways). This quasi administrative monopoly of policy evaluation, especially in crime prevention and safety matters, is seemingly accompanied on the part of policy-makers and officials by a solid mistrust of academia, who they feel are too quick to criticise public programmes. For their part, academic circles do not seem to be particularly keen on this evaluation activity. Only political scientists specialising in the analysis of public policies show some interest in it17, although not all of them. This outlook does not really retain the attention of those who use a bottom-up approach to study public policies, i.e. interactions between the street-level 15. Based on the report presented at the Bologna seminar on the French example by Anne Wyvekens (this volume and ref. cit). 16. A set of public policies addressing impoverished urban areas. It should be remarked that the task force in charge with this mission was not only composed of civil servants: for its better part, its preliminary work had been conducted by academics, notably Jacques Donzelot who has drawn upon this expertise to publish a book with Philippe Estèbe (1993). 17. See e.g. Lascoumes, Setbon, 1995; Kessler, Lascoumes, Setbon, Thoenig, 1998.

18

Evaluation_Safety.book Page 19 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

administrative officer and the client, or from the point of view of the latter. There exists a sort of scholarly consensus which estimates that what is undertaken by the administration in the name of ‘evaluation’ constitutes merely an internal audit, and that there is practically no genuine evaluation of public policies done in France, especially on safety and crime prevention. However, the Institut national de la statistique et des études économiques (INSEE, National Institute for Statistics and Economic Studies) has conducted a study on the trend in the relative situation of neighbourhoods where the politique de la Ville was implemented between the 1990 and 1999 censuses. This work by government statisticians is what best approximates an evaluative study. For the most recent period, among the many national observatories operating in the field of prevention and safety policies, those dealing with disadvantaged urban zones and above all drugs and addiction have published reports that are clearly in line with an evaluative perspective. We can still cite some specific studies, for example on the use of CCTV in high schools of the Île-de-France region (Paris and suburbs) but it is only to point out that their dissemination has been seriously curtailed by the instructions of the sponsors. Lastly, mention should be made of a certain number of studies, evaluative in nature, on the penal career of convicts sentenced to various types of penalties, after their release.

3.

An example of the meta-evaluation of a safety and crime prevention policy18

Queries by the Dutch Court of Audit regarding the performance of a crime prevention and safety programme, initiated in 2002, have led to a metaevaluation in three phases by the Social and Cultural Planning Office/ Institute of Social Research of the Netherlands (Sociaal en Cultureel Planbureau, SCP) and assigned to Van Noije and Wittebrood. These authors first collected some 150 evaluations on various aspects of this policy. They then selected those that they thought conformed to the evaluative principle defined in the classic report for the United States Congress prepared under Lawrence Sherman’s direction, and also with the criteria laid down by the Campbell Collaboration Crime and Justice Group for 18. Based on the report presented at the Bologna seminar on the Dutch case presented by Karin Wittebrood (this volume and ref. cit).

19

Evaluation_Safety.book Page 20 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

promoting the development of a quasi-experimental evaluation in the field of crime and criminal justice. They ultimately used the studies selected to review the different chapters of the government policy under scrutiny – law enforcement, developmental crime prevention, situational prevention, and lastly, systemic measures – while emphasising that all the studies necessary to review several aspects of the programme initiated in 2002 were not always available, and that it was necessary to encourage a still more systematic development of quasi-experimental evaluation. It is not the first time that the Dutch government has thus asked for a systematic review of the available evaluations19. The raw material has been provided thanks to the old habit of reserving 10% of the amount allocated to crime prevention and safety programmes for evaluation. Such a routine tends to show a certain practice of meta-evaluations, although, each time, the reviewers deplored the extremely inconsistent quality of the evaluations carried out.

4.

An abundance of evaluations and discussions on methodology20

In England and Wales, since the 1980s, evaluation of the crime prevention and safety programmes has been systematically carried out. What is more, since its return to power in 1997, the Labour party has combined its policy of crime and disorder reduction with an Evidence-based policy and practices (EBPP), which primarily rests on the accumulation of expert knowledge acquired from the systematic review of evaluations. In undertaking the latter, the standardised criteria developed in the aftermath of Sherman’s work at the University of Maryland and that developed by the group pursuing the work initiated by Donald Campbell, carry more and more weight. We can thus count the evaluations carried out: 앫 Home Office Crime Prevention Feasibility Study based on research using the case study method; 앫 Neighbourhood Watch Evaluation that has adopted a quasi-experimental approach on the basis of survey data; 앫 Community Policing Fear Reduction, also of a quasi-experimental bent; 앫 Priority Estates Project, quasi-experimental too; 19. See Junger-Tas, 1989; Polder, Van Vlaardingen, 1992; Soetenhorst de Savornin Lohman, 1997; Van Dijk, 1993. 20. Based on the report presented at the Bologna seminar on the Anglo-Welsh example by Tim Hope (this volume and ref.cit.).

20

Evaluation_Safety.book Page 21 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

앫 Kirkholt Burglary Reduction Project that focuses on an assessment of the effectiveness of programmes aimed at repeat burglaries; 앫 Safer Cities programme, which includes the sophisticated use of microeconometric methods by Ekblom et al. (1966), but also another one using qualitative techniques (Tilley, 1993b); 앫 Design Improvement Controlled Experiment (DICE), using a range of techniques including survey data; 앫 Communities that Care (CtC) Demonstration Projects for the Joseph Rowntree Foundation by a research group from Sheffield University; 앫 New Deal for Communities Programme (NDMC), employing a battery of indicators and analytical approaches; 앫 Crime Reduction Programme (CRP), combining some 80 evaluative studies based on the reduction of burglaries and remote surveillance through CCTV; 앫 Reassurance Policing. This abundance of works using a variety of methods and data enables a detailed discussion of evaluative methodology. This puts the British rapporteur in a position to identify the flaws in an evaluation limited to the ‘black box’ model, discuss the merits of Pawson and Tilley’s ‘realist experiment’ evaluation, show how the ‘quasi-experimental’ approach itself (Farrington, 2003) comes up against difficulties, to resolve which he proposes borrowing elements from micro-econometrics. The systematic use of evaluation in piloting prevention and safety policies can facilitate the discussion on the consequences of such an extensive mobilisation of scientific resources and their stowing to the goals of public policies. In this regard, Hope (this volume and 2009) suggests that such a situation could paradoxically lead, not to the ‘scientification’ of the politics, but to the ‘politicisation’ of science, if it is combined with a compelling aversion to critical results. There can be no contribution of science to politics except if each maintains its autonomy in relation to the other, and therefore its own qualities. That the evaluation of crime prevention and safety policy refers to a series of highly contrasting practices in European countries, which in other respects are quite similar to each other, highlights the difficulty of outlining a synthetic review. Such an endeavour should be able to deal both with the problems engendered by the lack of evaluation, and with those caused by its over-utilisation.

21

Evaluation_Safety.book Page 22 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

III. An ostensible evaluation without its risks The temptation is great to avail the benefits of evaluation without subjecting oneself to risks. There are two ways of doing this… The first consists of evaluating oneself, the second is to control the external evaluator really well so that he is practically forced into arriving at only positive conclusions.

1.

The charms of self-evaluation

Self-evaluation is the simplest. Thus, according to Smeets and Tange (this volume), the organisational reform of the Belgian police meant that the policing projects of the safety contracts21 were reserved for an internal evaluation undertaken by this very department. Wyvekens (this volume) gives an even more significant example of ‘the evaluation of public policies’ in France, an administrative activity governed by statutory instruments22, whose responsibility lies with the inspectorates attached to the various departments. In reality the reviewer is too close to the policy implementation to produce a genuine evaluation and these so-called ‘evaluations’ are, at best, internal audits. Sometimes, they are not even that: Roché (2005) cites the example of the ‘evaluation’ of the community policing initiatives by inspectors attached to the department of the Interior: he observes merely a long list of subjective judgements. With regard to the denunciation of self-evaluation, the specialists concur and sometimes in rather harsh terms23. From the inside, the risk is too great of mixing up the output of a programme with its effects on a population or a territory, for a whole series of wrong reasons. From this position, the access to internal data is so easy that the evaluator is tempted to use it as such24 without meticulously examining whether it really measures what has been done or the impact that this action could have had25. 21. Which later became a strategic plan for safety and crime prevention. 22. Decree no. 98-1048 of November 18, 1988 relative to the evaluation of public policies. 23. See e.g. Brodeur, 1998b. 24. Hence, according to Smeets and Tange (this volume), the Federal Ministry of the Interior obliges holders of security contracts to use the department’s internal statistics ‘in order to be objective’(sic). 25. They could possibly even be presented so as to give the impression of a positive result.

22

Evaluation_Safety.book Page 23 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

From the inside, it is difficult to get away from the logic of action: if the objectives were not implemented or if they were incorrectly implemented, that can easily be observed; but if they were, it will be difficult to envisage that the effect has not necessarily reached the target or that it differs from what was planned. In any case, if the internal auditor manages to discern the intended effects, he is in the worst possible position to see the unexpected, side- or displacement effects. In practical terms, this requirement has two consequences. Evaluation can be asked neither from an organisation entrusted with the programme’s conception or its implementation, nor from an organisation entrusted with the monitoring of a public programme, for example an inspection or an audit body26. Self-evaluation only produces illusory results, a surface evaluation. In her review of evaluations available in the Netherlands, Karin Wittebrood (this volume), concludes that none of those carried out internally by the sponsors of the prevention and safety programmes have even attained the minimum level of validity. To keep up appearances, a substitute is to call in private consultants. Smeets and Tange (this volume), as also Wyvekens (this volume), give multiple examples of the preference for this solution both in Belgium and France. By and large unrestricted by the principles of evaluative research, they are as quick as their sponsors would like them to be and their conclusions are not upsetting. In the Netherlands, Wittebrood (this volume) considers their technical level at times acceptable, at times inadequate, but critical conclusions are never delivered.

2.

Neutralising the external evaluator

It can easily be comprehended that self-evaluation (or the recourse to obliging private consultants) has all the appearance of an evaluation without incurring its risks. It seems more of a paradox to locate a comparable risk in an excessive ‘evaluation’. However, the systematic integration of an evaluative project in all crime prevention and safety programmes can lead to a type of domestication of the external evaluator, who becomes party to government contracts at the

26. Thus the evaluation scheme of the Chicago police reform was entrusted to a consortium of local university departments sponsored and financed by the State of Illinois. It was completely independent of the municipality and the police corps, who are in charge of conceiving and implementing the CAPS programme (Skogan, Hartnett, 1997a, b).

23

Evaluation_Safety.book Page 24 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

risk of losing some of his independence, somewhat like the journalist who is embedded in an army unit in action. There is no full guarantee that even the strict standardisation of evaluative research operations suffice to ensure against this danger. Methodological techniques can constitute an illusory defence that are brandished all the more readily as the other criteria of scientific production like peer reviews and the publication of procedures and results in an easily replicable format are sacrificed In this case, far from achieving the ‘scientification’ of political action considered by the initiator of the procedure, Donald T. Campbell (1969), as being the criterion of a good ‘open’ society, what is actually achieved is ‘politicisation’ of scientific activity. It is probably this risk, rather than the difficulties of the method, that deters many researchers from taking up evaluation work. But in which case, the recourse to evaluation will not really bring any added value, which presupposes accepting the risk of a negative conclusion by the evaluator, and also signifies that the latter himself is prepared to conclude negatively as well as positively. Hence, recognised evaluators (e.g. Hope, 2009) insist on a punctilious respect for autonomy and deontology peculiar to the scientific world in a discipline where the political stakes are extremely high, as insecurity has become less a problem needing a solution than a lever of governance (governing through crime).

IV. Evaluation criteria The weakness of all these shortcuts seeking the benefits of evaluation without incurring risks stem from a mistaken idea of evaluation.

1.

Internal audit and evaluation

Comparing the objectives with their implementation, determining the costs and comparing them with the initial previsions, describing the output, examining it in relation to the objectives, all these steps resemble an audit, a cost-effectiveness calculation (value for money) or programme monitoring, but they do not constitute an evaluation27.

27. See e.g. Sherman et al. (1997): a description of the implementation, its monitoring do not compare with an evaluation; see also Brodeur, 1998b or McKenzie, 2006, 342.

24

Evaluation_Safety.book Page 25 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

Confining the assessment to measuring what has been done is not an evaluation. Evaluation really begins with the determination of the outcome. An example28 will highlight the difference. The setting up of twenty CCTV cameras in the streets of a town constitutes an output, not an outcome: this is what has been done. On the other hand, achieving a 20% reduction in crime on the streets and minimising fear definitely constitutes results. Verifying that what had been planned has indeed been achieved is not yet an evaluation, even if this is a required preliminary step. Evaluation begins with seeking, not what has been done, but the effect that this action has had on the target. Often, however, it is wrongly believed that an evaluation has been achieved, whereas the substantiation was not pursued that far, but was limited to recording the outputs and not the outcome. This can be practised from a managerial perspective, which stresses that implementation must meet the objectives, also that expenses should be monitored, and which offers different techniques of verification. Paradoxically, this approach can also result from a strong administrative tradition whereby a central agency has the necessary resources to carry out inspections and verify whether the implementations correspond to the intentions of the politicians.

2.

Evaluation and its data

This clarification on the very nature of the exercise affects its materials: as a general rule, the internal data recorded by the concerned body is not enough for an evaluation29. The reason is simple: those records only catch events that came across institutional action. Having a burglary or a mugging figure in police statistics depends not only on the occurrence of a victimisation, but also on the propensity of the victim to lodge a complaint and that of the police to record it in a format that allows reckoning. One example will illustrate the difficulty: take an increase in the police recorded crime. Should this lead to the conclusion that the crime situation has worsened? Not necessarily. On the contrary, surveys can actually reveal that there has been a decrease in the number of victimisations. It is the propensity of the victims to lodge complaints that has gone up30 or their 28. Borrowed from the manual edited by Hough and Tilley (1998) for the Home Office for the purpose of guiding the local authorities in their implementation of the 1998 Crime and Disorder Act. 29. See e.g. Brodeur, 1998b. 30. This example is given by Wittebrood (this volume) in connection with the Dutch programme. Similarly Sherman’s (et al., 1997) example described the American experience.

25

Evaluation_Safety.book Page 26 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

recording that has become more systematic31. Relying on official records alone could lead to a mistaken interpretation of the observed trend. Moreover, counting does not always help to appreciate the prevalence of victimisation. For example, the increase in the number of recorded offenses can be due to the enlargement of the affected population or, on the contrary, to an aggravation of incidence – the number of victims has not increased but each victim has undergone repeat experiences. To appreciate the impact of a crime prevention programme, these two hypotheses have necessarily to be distinguished, diluted victimisation or concentrated victimisation. Often, only a survey will determine this. Lastly, the effect of victimisation depends greatly on the manner in which it is experienced. One of the French surveys32 came across victims who were not very traumatised – their mishap was due to their lifestyle and a slight change in it could easily prevent a repeat, for example not roaming in certain areas at certain times. Whereas others took it very badly, for although the incidents were only marginally more serious, the victimisation was linked to their living conditions: for example, they would have had to move from their present residence in order to be safe and they did not have the necessary resources. To evaluate the impact of crime prevention programmes involves distinguishing between these two types of experiences. Here too, simple institutional data is not enough, a social survey is imperative. Tilley (1995) also notes that institutional data can be presented in a biased manner – it is quite understandable but nonetheless misleading – so as to give the impression that the programme is effective, even if the results are indecisive. But using pre-existing institutional data is generally a cheap and (relatively) quick solution, whereas doing a survey is both costly and time consuming33. So what is to be done? Firstly, other pre-existing data from other sectors can be resorted to34 allowing cross-checking: the unpaid rents in public housing, statistics on school incidents or truancy … It is then possible to pre-position the surveys, for example by carrying out, within the framework of a regional crime observatory, regular surveys on victimisation, insecurity, self-reported crime. Even if certain areas have to be oversampled in order to gain specific knowledge about them, the cost will be less than starting a survey from zero, and above all it will be quicker. 31. 32. 33. 34.

Example in Hough, Mayhew, 2004. Zauberman, Robert, Pottier, 2000. Ekblom, 1998; Hough, Tilley, 1998. e.g. Hough, Tilley, 1998; compare Lagrange, Sina, 1998, although they deal with preliminary audits and consider that an evaluation should resort to other data.

26

Evaluation_Safety.book Page 27 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

Also it would be better to evaluate specific programmes and sanction the funds needed for a useful survey, rather than pretend to evaluate everything without sufficient data. It is here that the stipulation setting aside a percentage of the funds allotted to crime prevention policies for evaluation, as is the custom in the Netherlands, is very useful. Lawrence Sherman (et al., 1997) says it explicitly: they must make it feasible to carry out costly surveys which are required for a serious evaluation. Lastly, one should not wait for the programme to be implemented, or what is worse, completed before thinking about which data will be useful for its evaluation. Some data should be collected before starting in order to compare the ex ante and ex post situations. On the subject of data, one question is also whether quantitative data is alone valid for evaluative research. A priori, nothing counter-indicates that qualitative data should be used. Recently, at the Marc Bloch Centre in Berlin, in my conclusions at a political scientists' conference assessing public policies via the interactions of street level bureaucrats with their clients, I remarked that some of the ethnographic observations could have provided good evaluative material. Moreover, we know that Tilley (1993a, b) has used qualitative techniques in the evaluation of portions of the Safer Cities programme. Pawson and Tilley (1994, 1997) have argued in favour of a ‘realist’ evaluation that resorts to qualitative analyses, a view which has led to a disagreement with the promoters of the Campbell Collaboration Crime and Justice Group, and in particular David Farrington (2003). In truth, the discussion goes beyond the mere status of data: it is more concerned with the review of a ‘quasi-experimental’ method, which Pawson and Tilley criticise for focusing only on what works. These authors regard as evaluable only what has worked in a given context and in view of this context... on the basis of rather ethnographic observations. Whatever the outcome of this quarrel, it is not certain that it condemns the use of qualitative techniques in the evaluation35. In any case, measuring the impact of a programme entails not confining the approach to a simple audit or a management control, separating the evaluation from the running of the programme, and lastly, seeking the resources to generate data external to the programme under scrutiny.

35. All M. Q. Patton’s work (see especially 1980) constitutes a forceful plea in favour of a qualitative approach in evaluation. He defines the latter as: determining how a programme affects people (328).

27

Evaluation_Safety.book Page 28 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

V.

Measuring beforehand the substance of the action being evaluated

For all that, if the estimation of what has been done cannot be considered an evaluation, it nevertheless constitutes a mandatory first step in order to avoid measuring the impact of what did not really take place. One should therefore start by determining the substance of the object whose impact we are examining. Few spheres of public action are more subject to announcement effects than crime: there are talks and promises, but all these statements do not constitute faithful descriptions of what is really going to be undertaken. Failure due to the ineffectiveness of a programme has to be differentiated from that attributable to a nonexistent or uncompleted implementation. It is generally agreed that the preliminary phase of an evaluation should take into account the objectives, the inputs, the implementation and the outputs. It is the former that poses the biggest problem: generally, the boat is overloaded and a multitude of objectives publicised, sometimes without any clear relationship with the action taken36. In case of multiple partnerships, which is usually the situation, these aims can differ from one partner to another and sometimes they even contradict themselves37. There are some who feel it is necessary to distinguish declared from latent aims. This entails the risk of setting off on an exhausting race to determine the intentions. Would the task be easier when moving downwards from global policies to individual programmes? I have come to the conclusion that it is not worth dwelling too much on this clarification of the intentions: the auditor or the management controller would, on the other hand, keep at it. For the evaluator, it is the content of the action that is more important than the initial objectives. However, their consideration can, to a certain extent, weigh on the choice of criteria by rejecting those that correspond to a goal unrelated to the programme's rationale. Thus the Community Policing38 option implies the choice of a preventive or ex ante approach rather than a 36. Thus Brodeur (2000) notes the distance that exists between the prolixity of the discourse on Community Policing and the relative inadequacy of the concrete steps taken for its implementation. 37. To the extent of challenging any evaluation that wishes to focus on this contradictory multiplicity: Crawford, 1998b. Thus Breen (2002) and his associates, in their work on the evaluation of the justice system, note the lack of consensus over its objectives; they set about determining some by themselves, only to be disclaimed by their sponsors who, in their implementation ‘forgot’ those of these objectives that could have turned embarrassing, like impartiality or equality (Beauvallet, 2009). 38. See e.g., Skogan, 1990; Fielding, 1995; Rosenbaum, 1994; Skogan, Harnett, 1997a, b; Brodeur, 1998a, 2000.

28

Evaluation_Safety.book Page 29 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

repressive or ex post one. Deterrence through surveillance will be favoured over waiting for the offense to be committed before intervening. The evaluator would otherwise be completely mistaken if he chose statistics pertaining to crime control as a measuring tool, recorded offences for instance: he would estimate the programme by the yardstick of a goal that it does not pursue. The difficulty is critical in the case of social prevention programmes that only indirectly aim at crime and insecurity reduction, through other objectives, such as stable job access programmes or the reduction of poverty. By evaluating them on the basis of objectives they do not pursue directly, the opportunity is missed of verifying whether they attain their immediate objectives or not. There remains the difficulty of determining whether the success of the immediate goal has ricocheted positively on the indirect objective, namely reduction of crime and insecurity. Determining the resources is especially important for management auditors who want to measure the declared costs/outputs ratio from a managerial perspective39. This is less important, it seems to me, for the evaluator. In any case, it involves identifying – not always an easy task – the new or additional resource specifically related to the programme. Often programmes function by redeploying existing funds or by tapping into constant or relatively constant resources. In such a situation, the additional cost is marginal, at least in its quantifiable aspect, for its implementation can demand extra effort, voluntarism and imagination on the part of the personnel, which is all the greater as the additional means given them are rather meagre40. In the evaluation, determining what has actually been implemented constitutes, it seems to me, the crucial point of the preliminary phase41. This is how the programme will not be blamed for failing because of defective or, simply, non-existent implementation. It is a well known fact that in this field, much more is said than done, and that announcements often are a substitute for action. To determine the nature of the implemented action without too much difficulty, it would be really useful to have settled on the evaluation even before the commencement of the programme; it is also vital that the evaluator be independent of both policymakers and implementers; however, it is indispensable that he observe what is being done. This contradiction is not always easy to deal with: providing for the evaluation and its financing, having a central promoter 39. See e.g. Braga et al., 1999. 40. In this case, the launching of a new programme can bring in its wake – the evaluator should make note of this – a certain neglect of the customary tasks. 41. For example Brodeur, 1998b.

29

Evaluation_Safety.book Page 30 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

select the projects to be assessed and approving their auditors are the very preventative measures a national decision-maker – legislative or executive – could take to oil the wheels. Lastly, the preliminary phase of the evaluation ends with the estimation of the outputs. But this aspect is especially important for the programme management evaluator or for the auditor. By contrast, for the evaluator, delimiting the scope of the outputs specifically helps to distinguish them from the outcome, which constitutes the real criterion of evaluation42. Which is why the evaluator, contrary to the auditor or the programme management controller, cannot restrict himself to internal data; he needs a survey to constitute data that are external to the project and thus more apt to measure its outcome. Otherwise, according to Sherman (et al., 1997), the effort is subjected to evaluation, not the results. This is the stage where internal data is chiefly useful: to account for the implementation and its outputs. However the programme should not be trimmed down to a few indicators that would caricature it: only a comprehensive analysis can help to estimate the interventions' outcome43. Moreover, a description of the operations is not enough either, they have to be replaced in their proper context before the replicability of the observed results can be estimated. Hope (this volume) goes further and advises taking into account, not only the existence of an action, but also its intensity44 by calculating an intensity score45, borrowing the methodology from econometrics, an elegant way to measure the global substance of the action to be evaluated.

VI. Evaluative know-how A sort of minimal standard has been set comprising a before/after comparison, consideration of control groups and areas, and lastly, examination of the operation/impact relationship. It is moreover on the basis of this yardstick that L. Van Noije and Karin Wittebrood (2008) have selected 69 valid evaluations out of the 149 they had managed to collect in the Netherlands. In so doing they explicitly refer to the precedents set by Sherman et al. (2002), Campbell Collaboration Crime and Justice Group (Farrington, Petrosino, 2001) and the Scientific Model Scale or SMS (Farrington, Petrosino, 2001 ; Farrington, 2003). 42. 43. 44. 45.

e.g. Crawford,1998b; Hough, Tilley, 1998. See Pawson, Tilley, 1994. See also MacKenzie, 2006, 342. Time series analysis of programme impact.

30

Evaluation_Safety.book Page 31 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

Founded and for many years presided by David Farrington, professor of Psychological Criminology at Cambridge University, the Campbell Collaboration Crime and Justice Group, proposes to undertake a systematic review of the evaluative studies conducted on the effectiveness of criminological interventions and make it widely accessible. It thus intends to continue Donald T. Campbell’s (1969) princeps work, generally considered the promoter of evaluative research. We should note that these two major authors are both experimental psychologists: the quasiexperimental approach that they propose for evaluating crime prevention and safety policies owes much to the mode of reasoning and methods prevailing in their initial discipline. Farrington (2003) retains five criteria for assessing the methodological quality of evaluations; he attributes the first four to Campbell: 앫 statistical conclusion validity refers to the critical nature of the relationship between the intervention and the presumed outcome; 앫 the internal validity refers to the examination of the chain of causation between the intervention and the outcome; 앫 the construct validity refers to the adequacy of the outcome with the operational definition and measurement of the theoretical constructs that underlie the intervention; 앫 external validity refers to the possibility of generalising the causal relationship between the intervention and the outcome; 앫 lastly, descriptive validity relates to the adequacy of the presentation, in a research report, of the key features of an evaluation. Among all the scales pertaining to the methodological quality of evaluations, the most famous is the Maryland Scientific Method Scale (SMS) developed on the basis of the report prepared for the American Congress by Lawrence Sherman and his collaborators. It constitutes five levels : 앫 level 1 : the simple correlation between a programme and the measurement of crime, at a single point in time; 앫 level 2 : temporal sequence between the programme and the crime outcome; 앫 level 3 : comparative measurement of crime between two or more comparable units of analysis, one with and one without the programme; 앫 level 4 : comparative measurement of crime between multiple units with and without the programme, controlling for other factors likely to influence crime; 앫 level 5 : random assignment and analysis of comparable units to programme and control groups. But Farrington (2003) criticises it for not covering all the dimensions of validity retained by the Campbell Group. 31

Evaluation_Safety.book Page 32 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

The before/after comparison is obviously fundamental: without it there simply cannot be an evaluation. Four points at least deserve attention. First, evaluation should preferably be provided for before the commencement of the programme46 : the situation will be more easily observed ex ante rather than having to painfully reconstitute it ex post. Also, a sufficient number of criteria for this before/after comparison should be retained so as not to overlook unexpected effects. Such a precaution can be used for discerning the unintended effects more easily47: harassing dealers can reduce the impact of drugs in an area, but police intervention methods exasperate the youth so much that violence increases. Spreading a before/ after procedure beyond the sole implementation area of the programme allows for the identification of the possible displacement effects48 of crime: it may decrease in areas where the programme has been implemented but can surface in an adjoining area; it also allows virtuous contagion effects49 where investment in crime prevention is powerful enough to radiate in areas contiguous to its particular zone of intervention50. Specialists are highly critical of before/after measures that are not accompanied by the observation of control zones or groups where the programme to be assessed is not implemented. They even insist on the importance of having a pool of control zones or groups in order to neutralise the effect of a sudden crisis in one of them51. This amounts to an aspiration to move forwards from the rather primitive model of the ‘black box’52 to a method deemed quasi-experimental. For all that, the implementation of this procedure leads to massive problems. While it is possible to divide one neighbourhood or group into two and reserve one half for controlling, two difficulties emerge when one or several control areas or groups have to be located: i) it is difficult to find matching characteristics – two neighbourhoods seemingly similar in terms of their population and their problems, can nevertheless greatly differ in their histories and therefore their capacity for self-monitoring – and ii) another programme may be implemented in a control zone or population with effects similar to the effects resulting from the tested programme. However 46. e.g. Sherman et al., 1997; Brodeur, 1998b. 47. e.g. Sherman et al., 1997; but also Hope (this volume) with regard to the Kirkholt programme. 48. Hough, Tilley, 1998. 49. Displacement effects are more often envisaged than expansion effects (Hope, this volume). 50. e.g. Ekblom et al., 1996. 51. e.g. Bennett, 1991. 52. Said of an observation that considers the inputs and the outputs but not the intermediary mechanisms.

32

Evaluation_Safety.book Page 33 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

desirable it may be, comparability is often difficult to ensure. The aim of control zones or groups is to settle the issue: can the change observed be attributed to the programme or would it have occurred anyway despite its absence? But the functioning of this control is not automatic: thus the zero effect of a neighbourhood watch programme can simply be an expression of the similar levels of ‘social capital’ present in the experimental and control communities (Hope, 2009a). This type of difficulty obviously recedes with Tilley’s ‘realist’ evaluation: as it is no longer a matter of determining ‘what works’, but what has worked in a given context and in view of this context, it is obviously no longer necessary to find control zones or populations, nor expose oneself to the agony of this quest. From this perspective, it is not the programme that is being tested but the theory that underlies the actual implementation. More generally, the mechanical application of a quasi-experimental procedure does not help to steer clear of all the selection biases that Hope (this volume) has specifically drawn our attention to. Thus, a community can be selected because it seems to favour the programme to be implemented, but the effect observed can be as much due to the ‘social capital’ present in this community, as to the programme in operation here53. But specific deprived zones or groups can also be selected and then what is observed is merely their reversion to the mean. Which is why Hope suggests modelling selection effects by drawing inspiration from micro-econometrics. In any case, all these difficulties accord particular importance to the third phase of evaluation: there are three prerequisites to concluding as to the existence of an outcome, i) envisage and discard alternative explanations54, ii) on the contrary, explain how the programmes actually implemented have arrived at the results observed55 and iii) decide on the likelihood of this process. This is a crucial point where the evaluator’s know-how and experience prove useful. How about the downstream part of the evaluation, its utilisation? The monitoring of the programme should be considered separately from its replicability. 53. Even though a programme can at least have a positive effect on the mobilisation of the latent resources of the group or the area where it is being applied. It then becomes the intervening variable which produces the outcome, whereas the observation of a direct relationship between its implementation and the outcome is impossible. 54. Rosenbaum, 1986; among other things, there has to be data over a sufficiently long period of the earlier situation to discern possible tendencies that accounted for the before/after changes, even if the programme had not existed (Hough, Tilley, 1998; Hope, 2000). The statistical significance of the quantitative changes observed has also to be checked (Sherman et al, 1997). 55. See e.g. Tilley, 1993b; Hough, Tilley, 1998; Brodeur, 1998b.

33

Evaluation_Safety.book Page 34 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

As a mere post-implementation judgement – evaluation may only be of historical relevance to the programme, something which is not of particular interest to its directors and to the professionals. This is the real advantage of an impact study set up at the very outset: it is capable of providing information while underway, thus making it possible to adjust the tools56. But it is also expected from the evaluation that it indicate solutions promising enough to be replicable. The huge report that Lawrence Sherman was commissioned to write for the American Congress (1997), primarily pursued this objective, as did also the report on correctional measures by MacKenzie (2006). It is in fact the reason why evaluation should focus on certain programmes, which by their rigour and scope are likely, in case of positive evaluation, to be re-used on a wider scale57. Nevertheless, the exercise is a delicate one: to generalise a pilot-experience is not that straightforward; something that has given good results in a particular context can turn out to be less successful if transposed to other very different ones. Here, the selection biases can have a major impact, hence the importance of detecting and neutralising them before concluding as to the external validity of the programme under evaluation. * In the end, evaluation is a cumulative exercise. It is while evaluating that techniques may be improved. It is also while evaluating that practical knowledge of the impact of the different prevention measures is gradually mustered... knowledge that can always be revised, as is normally the case with scientific research, but knowledge that enables accounting for public policies and adjusting them as and when required. It is useless to try and evaluate everything, especially when experience and skills are modest. The outcome would be a pseudo-evaluation, based on impressions or trends and therefore doubtful. It is better to start modestly by selecting some specific programmes on which available resources and skills could be concentrated. Thus little by little, know-how will be developed and reasonably reliable diagnoses accumulated. Thereafter the process will have a snowball effect through trial and error. It should be reminded, however, that in the domain of evaluation the relationship between policy-makers and scholars is of a particularly delicate nature. Between the refusal of the former to approach the latter 56. This is what the Chicago police reform programme evaluation aspired to do. See, Skogan, Hartnett, 1997a, b. 57. Hence, Hough, Tilley (1998) advise reserving it for innovative, elaborate and costly programmes; see also the proposal made by the Sherman report (et al., 1997).

34

Evaluation_Safety.book Page 35 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

and, on the contrary, resort to a quasi-takeover, it is not easy to promote cooperation based on mutual respect of the independence of the two spheres. Without it however, evaluation can only be pretence. For all that, the effectiveness of a programme does not settle the question of the relevance of its location: the resources that are expended in one place could be in short supply in another where the needs are more urgent. Over and above all evaluation, we cannot shirk the task of reflecting on how the target-areas of interventions are prioritised, otherwise security measures in the long run will merely be the privilege of the well-off58. Lastly, crime prevention and safety policies are on the whole incapable of preventing the devastating effects of an accumulation of negative socioeconomic conditions on certain social groups or poverty-ridden urban zones59. These policies should not help to mask the absence of effective social and economic policies or worse, the continued accumulation of segregative decisions and practices. Without an effective reaffiliation policy they would merely be an illusion. This said, it would be interesting to find out how many (scientifically acceptable) evaluations have had a real change-inducing effect on public policies. On the subject of recommendations, here are some suggestions to anyone who would like to embark on the evaluation of crime prevention and safety policies: 앫 not to mix up evaluation – which applies to the impact of these policies on a target – with audit, programme controlling or cost-effectiveness calculations; 앫 to entrust the evaluation to a scientific body, competent and external to the institutions that are in charge of the programmes to be evaluated 앫 to respect the mutual exteriority of the policy-makers’ and the evaluators’ spheres; 앫 to plan the evaluation before the start of the programme; 앫 to provide data and know-how which are coherent with the nature of the evaluation.

58. Crawford, 1998a, b; also Hope (this volume). 59. Sherman et al., (1997) emphasise this fact repeatedly in relation to the United States.

35

Evaluation_Safety.book Page 36 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

References BENNETT T., 1991, The effectiveness of a police-initiated fear-reducing strategy, British Journal of Criminology, 31, 1, 1-14. BEAUVALLET M., 2009, Les stratégies absurdes. Comment faire pire en croyant faire mieux ?, Paris, Seuil. BRAGA A.A., WEISBURD D.L., WARING E.J., GREEN MAZEROLLE L., SPELMAN W., GAJEWSKI F., 1999, Problem-oriented policing in violent crime places: A randomized controlled experiment, Criminology, 37, 3, 541580. BREEN E., Dir., 2002, Evaluer la justice, Paris, PUF. BRODEUR J.-P., 1998a, (Ed.), How to Recognize Good Policing; Problems and Issues, Thousand Oaks, Sage. BRODEUR J.-P.,1998b, L’évaluation des programmes d’intervention sécuritaire ; quelques remarques sur les expériences canadienne et américaine, in FERRET F., OCQUETEAU F. (Dir.), Évaluer la police de proximité ? Problèmes, concepts, méthodes, Paris, La Documentation française, 13-22. BRODEUR J.-P., 2000, Les réformes de la police en Amérique du Nord: bilan des évaluations, Eines para l’analisi de la seguretat ; Estat de la recerca, Mollet del Vallès, Escola de policia de Catalunya. CAMPBELL D.T., 1969, Reforms as Experiments, American Psychologist, 24, 4, 409-429. COOK T.D., CAMPBELL D.T. 1979, Quasi-Experimentation, Chicago, RandMc Nelly. CRAWFORD A., 1998a, Crime Prevention and Community Safety. Politics, Policies and Practices, Longman. CRAWFORD A., 1998b, Culture managériale de l’évaluation et responsabilités ; quelques leçons de l’expérience britannique des programmes locaux de sécurité in FERRET F., OCQUETEAU F. (Dir.), Évaluer la police de proximité ? Problèmes, concepts, méthodes, Paris, La Documentation française, 51-82. DONZELOT J., ESTEBE PH., 1993, L’Etat animateur. Essai sur la politique de la ville, Paris, Ed. Esprit. EKBLOM P., 1996, Safer cities and residential burglary: A summary of evaluation results, European Journal of Criminal Policy and research, 4, 1, 22-52. EKBLOM P., 1998, Évaluation du programme Safer Cities ; impact et processus, in FERRET F., OCQUETEAU F. (Eds), Évaluer la police de proximité ? Problèmes, concepts, méthodes, Paris, La Documentation française, 27-50.

36

Evaluation_Safety.book Page 37 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

EKBLOM P., 1996, LAW H., SUTTON M., CRISP P., WIGGINS R., Safer Cities and Domestic Burglary, Home office Research Study 164, London, Home Office research and Statistics Directorate. FARRINGTON D.P., 2003, Methodological Quality Standards for Evaluation Research, Annals of the American Academy of Political and Social Science, 587, 49-68. FARRINGTON D.P., PETROSINO A., 2001, The Campbell Collaboration Crime and Justice Group. Annals of the American Academy of Political and Social Science, 587, 35-49. FIELDING N.G., 1995, Community Policing, Oxford, Clarendon. HOPE T., 2000, Counting on the Counter-Factual? Evaluation and Crime Reduction in Risk Society, Leicester, British Conférence of Criminology Conference. HOPE T., 2009, The evaluation of policies from a methodological perspective – Discussion, FP6, CRIMPREV, WP4, WS6, Universidade de Porto, Faculty of Law, Escola de Criminologia. HOUGH, M., MAYHEW P., 2004, L'évolution de la criminalité à travers deux décennies du British Crime Survey, Déviance et Société, 28, 3, 267-284. HOUGH M., TILLEY N., 1998, Auditing Crime and Disorder : Guidance for Local Partnerships, London, Home Office. JUNGER-TAS J., 1989, Meta-evaluation of Dutch Crime Prevention programs, Den Haag, Ministry of Justice, WODC. KESSLER M.C., LASCOUMES P., SETBON M., THOENIG J.-C. (Dir.), 1998, L’évaluation des politiques publiques, Paris, L’Harmattan. LAGRANGE H., SINA F., 1998, Réflexions de méthode sur les diagnostics locaux de sécurité, in FERRET F., OCQUETEAU F. (Dir.), Évaluer la police de proximité ? Problèmes, concepts, méthodes, Paris, La Documentation française, 91-114. LASCOUMES P., SETBON M., 1995, L’évaluation pluraliste des politiques publiques ; enjeux, pratiques, produits, Paris, CGP. LEIGH A., 2000, Policing in England and Wales: National Developments in Performance Management, Congrès internacional sobre les eines per a l’anàlisi de la sugeretat. L’estat de la recerca, Mollet del Vallès, Escola de policia de Catalunya. MACKENZIE D.L., 2006, What Works in Corrections. Reducing the Criminal Activities of Offenders and Delinquents, Cambridge, Cambridge University Press. MARTINSON R., 1974, What works? Questions and answers about prison reform, Public Interest, 35, 22-45. PATTON M.Q., 1980, Qualitative Evaluation and Research Methods, Newbury Park, Ca.

37

Evaluation_Safety.book Page 38 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

PAWSON R., TILLEY N., 1994, What works in evaluation research?, British Journal of Criminology, 34, 3, 291-306. PAWSON R., TILLEY N., 1997, Realistic Evaluation, London, Sage. POLDER W., VAN VLAARDINGEN F.J.C., 1992, Preventiestrategieën in de praktijk, een meta-evaluatie van criminaliteitpreventieprojecten, Justitiële Verkenningen, 2, 5-104. ROBERT PH., 1994, Évaluer la prévention, Archives de politique criminelle, 16, 53-70. ROBERT PH., 2003, The evaluation of prevention policies, European Journal of Crime, Criminal Law and Criminal Justice, 1, 114-130. ROBERT PH., 2009, L’évaluation des politiques de prévention et de sécurité examinée sous une perspective méthodologique, FP6, CRIMPREV, WP4, WS6, Universidade de Porto, Faculty of Law, Escola de Criminologia. ROBERT PH., LASCOUMES P., 1974, Les bandes d’adolescents ; une théorie de la ségrégation, Paris, Éditions ouvrières. ROCHÉ S., 2005, Police de proximité. Nos politiques de sécurité, Paris, Seuil. ROSENBAUM D.P., (Ed.), 1986, Community Crime Prevention; Does It Work ?, Beverley Hills, Sage. ROSENBAUM D.P., 1994, The Challenge of Community Policing: Testing the Promises, Thousand Oaks, Sage. SHERMAN L.W., FARRINGTON D.P., WELSH B.C., MACKENZIE D.L., (Eds), 2002, Evidence-based crime prevention. London/New York, Routledge. SHERMAN L.W., FARRINGTON D.P., WELSH B.C., MACKENZIE D.L., (Eds), 2006, Evidence-based crime prevention (revised edition), London, Routledge. SHERMAN L.W., GOTTFREDSON D., MAC KENZIE D., ECK J., REUTER P., BUSHWAY S., 1997, Preventing Crime: What Works, What Doesn’t, What’s Promising, Washington DC, NIJ. SKOGAN W.G., 1990, Disorder and Decline: Crime and the Spiral of Decay in American Cities, NY, Free Press. SKOGAN W.G., HARTNETT S.M., 1997a, Community Policing, Chicago Style, Oxford, Oxford University Press. SKOGAN W.G., HARTNETT S.M., THE CHICAGO COMMUNITY POLICING EVALUATION CONSORTIUM, 1997b, Community Policing in Chicago, Year Four: An Interim Report, Illinois Criminal justice Information Authority. SOETENHORST DE SAVORNIN LOHMAN J., 1997, New Forms of crime Prevention: the Dutch Experience, in HEBBERECHT P., SACK F. (dir.), La prevention de la délinquance en Europe ; nouvelles stratégies, Paris, L’Harmattan, 83-100. TILLEY N., 1993a, Understanding Car Parks, Crime and CCTV: Evaluations and Lessons from Safer Cities, London, HMSO.

38

Evaluation_Safety.book Page 39 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Europe

TILLEY N., 1993b, After Kirkholt – Theory, Method and Results of Replication Evaluations, Crime prevention Unit Series, Paper 47, London, Home Office. TILLEY N., 1995, Thinking about Crime Prevention Performance Indicators, London, Home Office. VAN DIJK J.J.M., 1993, The State of Knowledge in the Netherlands, in ROBERT PH. (Ed.), Crime and Prevention Policy, Eigenverlag MaxPlanck-Institut für ausländisches und internationales Strafrecht, 243263. VAN NOIJE L., WITTEBROOD K., 2008, Sociale veiligheid ontsleuteld: veronderstelde en werkelijke effecten van veiligheidsbeleid. Den Haag, Sociaal en Cultureel Planbureau. WYVEKENS A., 2008, Quand l’urbain rencontre la sécurité : la prévention situationnelle en France aujourd’hui, Revue de droit pénal et de criminologie, 9-10, 897-900. ZAUBERMAN R., ROBERT PH., POTTIER M.-L., 2000, Risque de proximité ou risque lié au style de vie ; Enquêtes et évaluation de la sécurité urbaine, Cahiers de la sécurité intérieure, 42, 193-220.

39

Evaluation_Safety.book Page 40 Tuesday, May 26, 2009 12:46 PM

Evaluation_Safety.book Page 41 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Belgium1 Sybille Smeets in collaboration with Carrol Tange

The question of the evaluation of safety and crime prevention policies arose along with the policies themselves, mostly since 1992, with the implementation of safety contracts (contrats de sécurité, veiligheidscontracten), emblematic of the fight against urban insecurity in Belgium, and constituting the basis of a contractual partnership between some cities or municipalities, two of the three Regions2, and the federal government. The latter, represented by the Ministry of the Interior, selects and funds policing and/or social crime prevention projects3 advocated by communities evidencing an urgent need to improve safety in their area (which need is mostly assessed in terms of recorded crime): grants are accompanied by strict instructions, including the required evaluation of the efficiency of the projects financed. Safety contracts have developed considerably since their early days, in terms of both the amounts of money funnelled through them and the number of cities and municipalities receiving part of the sums (103 as of now4), but there have also been many reorganisations, the main ones being the elimination, in 2002, of police projects5, and the transformation, in 2007, of the safety contracts into Strategic action-plans for safety and prevention, henceforth signed for 4year periods.

1. 2. 3. 4. 5.

Translation by Helen Arnold, revised by Renée Zauberman. The Brussels-Capital Region and the Walloon Region. Which mainly target vulnerable or marginalised social groups or neighbourhoods. For a total of 589 cities and townships. Along with the development of a crime prevention policy, an unprecedented development of policing policies occurred in the ‘90s. Starting in 1991, a series of measures were taken to improve the functioning of the police, especially at the local level. The modernisation and valorisation of local police, along with evaluation of the impact of measures taken in that framework, also took the specific form of the implementation of safety contracts. The reform of police services (1998 Act) and the elimination of the policing projects of safety contracts led to the gradual disappearance of external evaluation of the police, replaced by self-assessment focusing on internal functioning and management of the police corps.

41

Evaluation_Safety.book Page 42 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

This means that evaluation policy, impetus for which comes from the federal government, has mainly focused on safety contracts. Whereas the other governmental levels – local, regional and linguistic community – do introduce and fund prevention and/or safety policies, these are more rarely evaluated and are only exceptionally governed by clear instructions. When administrative and/or internal evaluations do exist, both their form and their content are actually largely inspired by federal evaluations. Our initial assumption is that evaluation, as opposed to scientific research strictly speaking, always involves a political dimension: first, it must answer questions aimed at producing political action6, and is intrinsically tied to political decision-making as to what are deemed high priority collective or individual needs for the implementation of the programs or projects assessed, but also for the allocation of public resources7. Secondly, evaluation is intended for all sorts of “constituencies” and “interlocutors” who may be competing as to how to interpret and use the findings produced8. The concerns and priorities of policy-makers, agencies commissioning the evaluation, actors and beneficiaries of the programs and projects evaluated, the media, scientific communities and citizens themselves are neither necessarily coherent nor necessarily convergent. Last, not only are there many constituencies, but also a wide range of evaluation methods susceptible of use (particularly in terms of methodology9). These are also subject to negotiation, reciprocal adjustments between evaluators and sponsors, judgments and challenges that extend far beyond scientific or technical considerations on the internal or external validity of the evaluation. These choices reflect both the objective – explicitly formulated or not – that one wishes evaluation to achieve, and the constituency or constituencies viewed as priority targets10. The elements left in the background are as revealing, in this respect, as what is set in the forefront. We have chosen, here, not to confine ourselves to scientific evaluations, and within these, to what may be described as systematic (or even quasiexperimental) research. To the contrary, evaluation is taken in the broadest sense of the term here, barring which we might well arrive at the

6. 7. 8. 9.

Lacroix, Tange, 1997, 7. Greene, 1994. Greene, 1994. Methodology is taken here not only as the methods and techniques for data collection and analysis, but as including epistemological and theoretical positioning (Lacroix, Tange, 1997, 18). 10. Lacroix, Tange, 1997, 18.

42

Evaluation_Safety.book Page 43 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Belgium

conclusion that ‘there is no evaluation’ in Belgium11. To the question, are the pieces of research under review really of evaluative nature, that is do they really confront the objectives, inputs, implementation and outputs of the policy or programme under scrutiny, with its outcomes, including collateral or unexpected effects?12, we would be obliged to answer in the negative in most cases. However, since the idea was to appraise existing efforts, we started from the following question: what is defined, or defines itself, as evaluation? This point of departure seemed particularly relevant, since the idea here is to evidence what, concretely, is designated in Belgium by the term “evaluation”. We divide existing evaluations into two groups, each with its own specific features. The two groups differ mostly as to the respective institutional positions and roles of evaluators, which are not devoid of consequences on the content and goal of their evaluations. We shall begin with “internal” evaluations, which include both administrative evaluations of safety and crime prevention policies, done directly by and for the funding agencies, and secondly, internal evaluations by the agencies receiving those funds. We shall then go on to discuss scientific evaluations (evaluative research) conducted by external and independent research centres which may or may not be attached to universities, and are commissioned by the same funding authorities.

I.

“Internal” evaluations or the predominance of summative evaluations

Internal evaluations differ most from scientific evaluations by their goal, which is mainly summative (or sanctioning)13. They aim at establishing an assessment of the programme or project implemented, and at judging its value. They are therefore essentially aimed at decision-making with respect to the continuation, extension, or reduction of those programs or projects.

11. As some French researchers and practitioners do (see the report on France). 12. Commissioning letter. 13. Patton, 1980.

43

Evaluation_Safety.book Page 44 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

1.

Administrative evaluations

Administrative evaluations are conducted by services attached to the funding authorities, in this case the Ministry of the Interior14. They form a non-negligible part of the monitoring of cities and municipalities compliance with those minimum obligations attending safety contracts, their respect of the clauses of the contracts, and their proper use of the funds. Relatively few data are used: the content of the contracts themselves, some administrative information on budgeted expenditures and the personnel employed, and annual visits to view the projects at work. The main objective here is to enforce accountability and to evaluate the effectiveness15 of implementation, with evaluation of efficiency16 and efficacy left to internal evaluations of the projects. Theoretically, a negative administrative evaluation is sanctioned by the withdrawal of, or cuts in, funding. This type of evaluation therefore serves mostly in negotiating safety contracts with the cities and municipalities involved, especially for funding and adding projects to contracts. Administrative evaluation reports reflect this purpose: they display tables or pie charts revolving around figures summarising the number of contracts and the list of cities and municipalities, the number of ‘jobs’ created, the effect of each part of the contract on spending, per cities and municipalities or Region and for the federal ministerial department involved, along with a theme-based typology of projects. Exceptionally,

14. Until 2002, the General Police of the Kingdom (Police Générale du Royaume) was in charge of evaluating the policing projects of safety contracts, with the Permanent Secretariat of Crime Prevention in charge of the social and preventive aspects. In 2002, the latter became a part of a Department of Safety and Prevention Policy (Direction générale Politique de sécurité et de prévention), in charge of the monitoring and evaluation of all of the projects included in safety contracts. It is noteworthy that the advisers, appointed as evaluators, are often young academics with no experience in research or evaluation. 15. Defined as the appropriateness of the practices actually implemented to those planned (Laval, 1990). 16. Defined here as appropriateness of resources to the results, with special focus on the question of whether, in achieving the objective, the best possible use was made of the resources invested (Laval, 1990).

44

Evaluation_Safety.book Page 45 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Belgium

they may include police statistics and data from the Security Monitor17. The latter, a qualitative survey of the population, is in fact depicted as a sort of evaluation of safety policies in the broadest acceptation. Thus, in spite of some methodological limitations18, the Security Monitor has been repeatedly used as an element in evaluating citizen satisfaction with the police and the extent of sense of insecurity related to those kinds of crimes assigned high priority by federal and local authorities.

2. a)

Internal evaluations, or self-evaluation of safety contracts From evaluation to diagnosis (audits)

In theory, administrative evaluations must be completed by internal evaluations, performed, since 1996, by specially appointed municipal employees (internal evaluators). These evaluations, the required minimum for safety contracts, are officially intended to check on the implementation of projects, their impacts, and their efficacy with respect to two goals, in particular: decreasing sense of insecurity/fear of crime and preventing petty crime. Aside from these extremely general indications, the only specification is that for the sake of objectivity (sic) these evaluations be based on official crime statistics19. To this day, however, internal evaluators are in a difficult position: they are employed by the city or municipality, but paid (often part-time) by the 17. The Security Monitor takes the form of a questionnaire established by the federal police and administered by a private firm, by telephone, to a sample of the population (40,000 respondents for a 10 million population) randomly selected from the phone book, representative of the Regions and types of municipalities. The local Monitor focuses mostly on municipalities having underwritten a safety contract. Five themes are broached: neighbourhood problems, victimisation (including notification and reporting), sense of insecurity, contacts with police services, and functioning of the police. It has been administered six times since its inception in 1997. Data and methodology may be found on www.poldoc.be/dsb.htm. 18. These include, with no pretension of being complete, the subjective nature of the questions, the limited choice of answers (none of the categories proposed refers to anything other than offending and anti-social behaviour, for example), the biases inherent in surveys by phone, the limited representativeness of the sample (for instance, the elderly – over age 50 – are over-represented in comparison with the group aged 15 to 35; this actually has to do with the biases entailed by phone surveys), the decision, in local watches, to focus on cities and townships with safety contracts, the absence of contextualisation, the fact that local users rarely participate in the construction of the questionnaire, the difficulty in establishing comparisons with police statistics. 19. Official instruction from the Ministry of the Interior, dated September 19, 1996, pertaining to internal evaluation missions. Unpublished.

45

Evaluation_Safety.book Page 46 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

Ministry of the Interior, they must assess the locally developed policies, for the benefit of the local authorities, but also for the benefit of the funding agencies. On the basis of that evaluation, the latter may decide to renegotiate the contract, with cuts in its budget, or to eliminate some projects. Wearing these “two hats” is not devoid of impact on the content of internal evaluations. In practice, most of them merely inventory the actions under way and, like administrative evaluations, evaluate the effectiveness of the projects, of the personnel hired and of spending. In some cases, these evaluations also contain figures (often in appendices) taken from police statistics20, the local Security Monitor, the many reports written by field workers or, more rarely, small, very local, restricted surveys of residents’ satisfaction or their perception of safety problems. In 2000, to standardise these evaluations and enable comparison between safety contracts, the Ministry of the Interior drew up a series of guidelines for evaluation, many in questionnaire form, containing highly factual items further accentuating the administrative dimension and the analytic weakness of internal evaluations. In 2005, cities and municipalities were offered a methodological guide for a local safety audit (LSA)21. It recommends that the “audit” part be based on three items: an objective description of safety problems (police and court statistics, records of local administrative sanctions as an indicator of “antisocial behaviour”), a subjective description of safety problems (the local Security Monitor, any existing victimisation surveys) and a SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis22 of the policy implemented at the local level. In this context, the audit should focus on those federal priorities relevant to safety and prevention contracts, when identifying problem areas. These include property offenses, techno-prevention, anti-social behaviour, fighting drug abuse, violence, and the protection of some 20. In some cities and municipalities, attempts at local scanning (miniature surveys of residents by local police officers, for instance) and crime mapping (mapping of acts recorded by the police) have been developed. They remain methodologically questionable, and are not available to anyone outside the police corps. They serve operational police work (focused patrolling, for example), and only rarely contribute to evaluation. 21. Direction générale Politique de sécurité et de prévention, 2005. 22. Strengths: the strengths of an organisation or project enabling it to cope with any “threats” from the surroundings. Weaknesses:... the weaknesses and points requiring improvement in an organisation/project so as to meet, for instance, the new demands imposed on it by the environment. Opportunities:... factors facing the organisation/ project, and which may well result in the development or improvement of the organisation/project. Threats:... elements potentially constituting a threat for the survival of the organisation/project (Direction générale Politique de sécurité et de prévention, 2005, 68).

46

Evaluation_Safety.book Page 47 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Belgium

vulnerable groups in the population... Once the high-priority occurrences have been detected, strategic objectives should be formulated so as to manage projects and implement action correspondingly23. In 2007, when safety contracts were turned into strategic plans, the LSA became compulsory, as did the proposed guidelines, replacing internal evaluation in its strict sense. The obligation to compose a LSA affected the way the work of internal evaluators was reoriented. The stress, in the guide, on police statistics and Security Monitor data, along with the choice to focus evaluation on some criminal phenomena (rather than on projects) obliged internal evaluators to adjust constantly, and to do some “fiddling around”24 so that the audit would not be derogatory for their community, especially with respect to the many projects of a more social nature (community houses, street workers and various sorts of mediation) for which the evaluation criteria and focuses are hardly relevant. In recent years there has been a spate of local surveys of satisfaction, inserts on “good practices” or “testimony” (usually on innovative or wellfunctioning projects, even if these are not representative of the strategic plan as a whole), a juxtaposition of entry points to evaluations (by occurrences, projects, budgets, neighbourhoods) as well as one great novelty, the use of other socio-economic, demographic and urban development data, mostly in order to relativise crime figures. Similarly, the very aim of the LSA often forces internal evaluators out of their isolation, compelling them to call on other partners or key-informants to form an editorial committee or to collect the required information. As a rule, although the LSA is perceived, in most communities, as increasing the workload, in those cities and municipalities with sufficient financial and human resources, and whose prevention policy receives support from the local authorities, it also has an enriching effect on evaluation, especially in that it multiplies and crosses the references to which internal evaluators may compare the local situation. Having said this, another observation must be made: the content and quality of LSAs vary considerably from one community to another, depending on available resources and on the degree of integration of the funds and the levels on which safety policies are steered. When such integration is absent, only the funded initiatives are evaluated, with no 23. Direction générale Politique de sécurité et de prévention, 2005, 69-70. 24. As Poulet (1995, 20sq) points out, saying that actors ‘fiddle around’ with the available material does not mean they have no idea of the objectives pursued, but rather, that those objectives may be modified on the way..., that such changes of orientation are not completely under control, since each actor takes what the others do into account, and must continually adjust... and makes arrangements using elements borrowed from various contexts, each of which has a definite ‘logic’... which limits the uses to which it may be put.

47

Evaluation_Safety.book Page 48 Tuesday, May 26, 2009 12:46 PM

.

Evaluating Safety and Crime Prevention Policies in Europe

consideration for other projects and their possible effects; a number of small, short-term evaluations are done, often omitting the data available at other levels of administration, and especially those that provide contextualisation. Other difficulties, mentioned by the internal evaluators themselves, have to do with the timing – annual – of LSAs, too short for indepth analysis, and also, the frequent turnover in personnel working on safety contracts, including internal evaluators, which prevents the build up of experience required for follow-up and for familiarity with the projects. b)

Limited utility

Theoretically, internal evaluations are intended for use in negotiating contracts (that is, funding for them) and for defining new projects. In practice, they only serve the former objective. The choice to give greater importance to the LSA was intended to improve internal evaluation, since its goal is much broader than the a posteriori evaluation of existing schemes: the diagnosis, by definition, is taking place upstream of the action to be decided upon, and should be used to clarify or define actual problems. One of its main objectives, then, is to sense citizens’ expectations, to identify the problem (the symptoms), and in short, to define or, if necessary, to reorient local safety policies. In practice, the LSA replaces internal evaluation without breaking with the way the latter is conceived and used, and therefore serves the same goal, at least at the federal level. This is in fact criticised by internal evaluators themselves. On the one hand, it is incoherent to prefer a broad-based tool such as the LSA which should be used for defining guidelines for local safety policy, but also for the quality of local life, and to restrict its use to the evaluation (and funding) of programmes aimed at fighting some types of crime viewed as top priority by the funding agencies. On the other hand, there is much disappointment about the lack of federal evaluation based on compendiums and cross-analyses of the various data and evaluations produced by LSAs. LSAs are thus only used singly (in each community) and not as a tool for overall management, or for the circulation of “good practices” in evaluation, or for their improvement. At the local level, some cities and municipalities – usually the larger ones – perceive the value of this new tool, especially as a means of political communication and publicity (for brochures, press conferences, presentation to the local council, and so on) addressed to residents, their opponents, and the higher governmental authorities, and occasionally as a decision-making aid for locating areas and actions requiring priority attention. In these communities, LSAs have been useful in establishing or enhancing the legitimacy of local safety policies by producing a cohesive,

48

Evaluation_Safety.book Page 49 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Belgium

coherent image of a scattered series of projects which, taken individually, rarely possess high visibility outside the neighbourhoods where they operate. In this sense, LSAs offer a surplus value, at least for these cities and municipalities, which was not true of the “old-fashioned” internal evaluations.

II.

Evaluative research

Starting in 1985, the Belgian government, through the minister of the Interior, launched a vast scientific research programme aimed at universities. Whereas tenders were mostly invited for research on the police at first, the introduction of safety contracts was to extend the program to scientific evaluation of crime prevention and safety policies. The stated objective of the research programme, and of evaluative research in particular, was both the improvement of the organisation and operation of the initiatives and projects evaluated, and to provide scientific means and methods for improved management of crime prevention [and] to legislate knowledgeably in specific fields [policing and crime prevention]. Last, research was to be conducted so as to provide results liable to be, directly put into practice25. The objectives are still the same today. Two periods may be distinguished, however. The first, from 1991 to 1999, in which evaluative research were essentially formative, aimed at detecting the weaknesses in programmes and projects so as to improve them. The second period began with the change in the federal government in 1999, with a tendency to prefer research focusing on findings directly transposable into practices and/or policy orientations, and which ultimately dropped any evaluative approach, even when research was still nominally evaluative.

1.

1991-1999: cross-sectional evaluative research on processes and operation

At first, scientific evaluation, representing almost 75% of all research commissioned by the federal government during this period (over one hundred projects) were mostly entrusted to independent research centres, usually academic, mostly composed of criminologists, sociologists, and/or legal specialists.

25. Ministère de l’Intérieur, 1991.

49

Evaluation_Safety.book Page 50 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

Until 1999, evaluative research primarily focused on processes and operation: it examined the implementation of programs and/or projects so as to assess their facilitating or obstructive elements, or again, their overall coherence. Half focused on the effects or the impact of the action, the actors or the project on their environment (and especially sought to answer the question of problems and biases they generate). In some cases, this research also evaluated the relevance26 of these projects and programmes. With few exceptions, most of these evaluations are cross-sectional and focus on themes rather than on individual projects: they assess a series of identical projects across the national or regional territory. Comparisons are concerned with similarities or differences in the effects, impact or processes implemented. These are also evaluated with respect to their particular, specific objective (improved police-citizen relationship, better reception of victims, (re)integration of local residents into the labour market, fewer school drop-outs, coordination between projects, and so on) and are not intended to measure the effects of projects or programmes in terms of reduced crime or sense of insecurity. The majority (2/3) of evaluative research produces large amounts of empirical material, mostly based on qualitative methods (participant observation and/or semi-structured interviews). The most frequent themes include the local police, already the focus of police reforms, especially with respect to funding for the police projects in safety contracts: relations between the police and the public, and more specifically with youths, victims, immigrants, or drug users; the organisation of the police department; community policing; collaboration between police forces, and police statistics. Evaluations of safety policies, strictly speaking, focus on local, pilot prevention projects, school drop-outs, the coordination of safety contracts, the new field-work and managerial functions created for and by those contracts, and hooliganism27. During this period, researchers were given some leeway, especially with respect to methodology. This is evidenced in the composition of the research supervision committees – mainly other academics – and the effort at disseminating the results of evaluative research through a policy of systematic publication of books by the Ministry of the Interior and of articles in scientific and/or professional journals by the researchers themselves. Nonetheless, the objective of aiding decision-making and the importance of the Ministry of the Interior in commissioning the 26. Defined here as the linkage between needs (related to the original situation) and the objectives, means, resources, and participants deployed in the programme (Laval, 1990). 27. Ponsaers, Janssen, 1993, 39-79; Tange, 2002; Tange, 2003.

50

Evaluation_Safety.book Page 51 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Belgium

evaluations considerably limited the compass of evaluative research: some themes, although sensitive, were excluded28.

2.

1999-2008: A smaller range of subjects, internalisation of research, and reduced funding

After the 1999 general elections, invitations to tender for evaluative research funded by the ministry of the Interior called for increasingly “beleidsrelevant”29 evaluative research. These are both less ambitious, more operational, responding to the government priorities of the moment, and partially performed by private firms (consultancies and auditing firms) or by governmental agencies30, to the detriment of independent research centres, in the belief that the first two are better able to formulate policy recommendations31. Also, when evaluations are assigned to independent research centres, new disciplines – psychology, education, human resources, management, engineering, and computer science – are added to the disciplines previously involved in such research, further indication of the wish to give preference to more application-oriented and/or more technical approaches. This use of a multitude of types of expertise is often done to the detriment of interdisciplinary approaches. Evaluative research has also suffered budget cuts, which have affected timing and staff: six or even three-month research contracts abound, with the equivalent of a single full-time worker, whereas the norm during the previous period was at least one year, often renewable, with two full-time positions. Simultaneously, calls to tender also prefer partnerships between agencies, especially between Flemish and French-speaking universities, 28. Evaluation of policing rarely focuses on the gendarmerie, on criminal investigation agencies with the Prosecutor’s office, or criminal investigation in general, any more than on relations with the police hierarchy or police deviance. As a rule, traffic safety plans, although of great political concern, sense of insecurity/fear of crime and victimisation were not evaluated either until 1997. Last, although some youth support programmes are funded by the federal government, most are not evaluated. 29. The term, used by the federal government, may be imperfectly translated as immediately ‘relevant’ or ‘interesting’ for policies. 30. To mention a few, the criminology department of the National institute of forensics and criminology, the Crime policy service (Ministry of Justice), the federal police department and Department of Safety and Prevention Policy (Ministry of the Interior). Characteristically, since the police have been reformed, the police manage studies of a number of aspects of their own activity, previously examined by researchers not connected with that institution. This is true both of its (quantifiable) resources and activities and views and judgments thereon, and of safety problems in general (population surveys and other forms of scanning). 31. Tange, 2002, 56-57.

51

Evaluation_Safety.book Page 52 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

but also between public and private, or internal and external agencies, which further splits up the available resources. Although researchers are still given some latitude, this situation does nonetheless affect methodological options. Instead of long-term empirical studies, preferences go, first, to quantitative evaluations, for which it is easier to cross existing data rather than to produce new ones, and secondly, to restricted qualitative methods (focus groups, joint academic/ professional seminars, and so forth). Actually, the title ‘evaluative research’ now covers a series of studies which, unconcerned with impact and often heedless of the context, tend rather to establish inventories of means, existing, available, or requiring development, or to propose tools and methods for proper management, data analysis or policy planning (for establishing policing or safety priorities)32. In this context, the subject matters of “evaluative” research are gradually modified, to focus primarily on management and proper administration (content of training, budgets, time management, management, technology, legal adjustment, and so on) and based on relatively normative, technical or comparative approaches that provide little information on actual practices and aim at meeting the desire of the authorities to define, or actually, to provide a legal framework for those practices. It is typical of this trend that internal research agencies play a growing role in the formulation of invitations to tender (especially in terms of research subjects and methodology, and occasionally, ‘intended results’), and in the selection of evaluation providers, when they are external. This accounts for the redefinition in invitations to tender, often less based on the findings of prior research than conforming to the new political or internal orientations of these agencies. Similarly, research supervision committees are particularly receptive to ‘administrators’ of these internal agencies, to the detriment of academics, now definitely a minority. Consequently, circulation of findings is much more restricted, causing research to become invisible33: the policy of systematic publication has faded away, and some researchers are prohibited from publishing their findings, owing to their use of “confidential” databases, or to their methodology or technique, which requires “some discretion”, or quite simply because they are for purely internal use.

32. Prospective “evaluations” have also cropped up, using some predictive methods or resorting to experts so as to establish a picture of national safety, for the definition of future policy orientations. See Ponsaers, 2005. 33. Tange, 2002, 63.

52

Evaluation_Safety.book Page 53 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Belgium

3.

Use of the findings

Before 1999, two basic concerns (which balance out, more or less) may be evidenced in evaluative research: on the one hand, the uncovering and clarification of poorly known, complex situations, with the help of this research, and possibly, at the same time, the provision of tools enabling administrative bodies to continue the work themselves; on the other hand, the development of new policy and practical options, based in theory on clarification, for the ensuing monitoring and evaluation by the administrative bodies (administrative evaluation). In this context, the purpose of evaluative research is clearly intended as the redirecting and improvement of existing programmes and projects. Secondarily, academic evaluators also participated in (administrative) evaluation committees and in negotiations on safety contracts. Consequently, evaluative research also served for the renegotiation of contracts (summative evaluation). In spite of these objectives, the results of scientific research were rarely taken into account (or even read) by the federal or local authorities in charge of the projects, although every attempt was made to circulate the findings. In some cases, some projects (especially pilot projects) were renewed before their evaluation was completed. After 1999, invitations to tender concentrated on research susceptible of providing results directly transposable into practices and having little to do with an evaluative approach, in spite of their title. In practice, although research findings were put to use more systematically, they mostly served either for post hoc legitimisation of previously adopted political options, or as support for the operational development of those options. In this context, teams of researchers, both external and internal, private and public, are asked to act as experts in survey methodology, data analysis (especially quantitative data), normative frames of reference, for the benefit of the real operators of these tools, that is to say administrations and police agencies. While evaluative research has always been aimed at helping decision-making, we have gone from a patent desire for improved decision-making based on understanding to a determination to gain knowledge – and perhaps self-knowledge – the better to justify (oneself).

III. What is most lacking: measurement of impact on crime and sense of insecurity/fear of crime An assessment of evaluative research commissioned since the early ‘90s necessarily points to the absence of evaluations aimed at measuring the

53

Evaluation_Safety.book Page 54 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

effects of programmes and projects with respect to the two main objectives of prevention and safety policies: reducing crime and reducing fear of crime/sense of insecurity. Moreover, while the question of the impact of programmes or projects on the environment or on actors was central to evaluative research, especially before 1999, evaluations tended instead to assess the process or unfolding of an initiative, and possibly its reception by actors and beneficiaries, an approach which accounts for the predominance of qualitative approaches. We have several leads that may explain these choices, but make no claim to be exhaustive. The first explanation has to do with the Belgian political context. Safety policy has been developed by successive layering, and with a sense of urgency, in response to highly publicised events, relegating the fundamental issues to the background. This trend definitely affected conceptions of evaluation as well as research conditions, including subjects, timing, or other factors such as the choice of preferred methods. Also, attempts to respond rapidly to citizens’ alleged concerns about safety are made to the detriment of an analysis of needs and of evaluation performed previously to the projects' implementation, thus precluding any before/after comparative approach. But at the same time, the crisis in citizens’ confidence in public institutions, and in the police in particular, creates the need to make reliable empirical data available rapidly, which need brought about the first programme of invitations to tender for research. The preference given to evaluations of processes over qualitative or quantitative measurement of the impact of policies also originates in the objective assigned to these evaluations: providing scientific backing for the policies developed, but without questioning their cogency. Crime and public safety are indeed major electoral issues, dealt with differently in the various party programs. However, like the Belgian political system, which favours coalition governments, public policies on safety represent compromises between a more right-wing tendency, which prefers to allocate resources directly to the criminal justice system and the police, and a socialist tendency, which invented the safety contracts; secondly, between the Flemish, attracted more to the Anglo-American theories and approaches, and the French-speakers, whose preference goes to the socialminded approaches of French urban policies. Evaluation policy reflects this compromise: it is itself a compromise between, on the one hand the wish to evaluate only with crime reduction in mind, as measured by police statistics, and on the other hand the wish to stress qualitative evaluation based on the word and experience of actors and beneficiaries of the programs or projects. Until 1999, the compromise took the form of a 54

Evaluation_Safety.book Page 55 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Belgium

separation between quantitative "evaluation" limited to internal evaluations and qualitative evaluation left to the academics. At the same time, in political discourse, the overall objectives of safety contracts are set aside, and the focus is placed on bringing political institutions closer to citizens, so as to win back their trust. In the end, it is these objectives that have come to be of scientific evaluation. Until 1999, this lack of evaluation of the results of safety contracts was not really a problem for the government, which legitimated its policy by the claim that what counts is “doing something” and making that “something” visible, and secondly, by the fact that these policies were a continually growing source of employment. But the change of government in 1999, along with the commitment to rationalise resources, justified, in particular, by the huge sums swallowed up in reforming the police agencies, was to considerably toughen negotiations between supporters and opponents of local safety policies34. After the 1999 general elections, the efficiency of prevention policies were seriously called into question, especially since the minister of the Interior was right-wing. In this context, the lack of scientific evaluation, in the strict sense, led the way to divergent, partisan interpretations of the available quantitative data (especially of crime statistics), and sounded the knell of qualitative evaluative research, without however leading to any other forms of evaluation. The drastic budget cuts in the research program commissioned by the Ministry of the Interior, leading to changes in timing and cuts in the staff paid by the ministry, partially account for this situation: systematic scientific evaluation requires considerable spending, to which the government is no longer agreeable. But part of the explanation has to do with external and independent research centres themselves. For these centres, in fact, evaluation, strictly speaking, is still viewed as a “non noble” research approach, especially in criminology, where there is an opposition between "state criminology", “inside the system” and serving criminal justice policy, and "academic criminology", “outside the system”, claiming to be independent of the latter35. So that in the early ‘90s, many research centres agreed to go along with evaluation provided they were not “compromised”, particularly in respect to the goal of crime reduction, which they had been quick to criticise. Under the disguise of evaluation, then, these centres took advantage of the money offered by the Ministry of 34. Opposition between the various components of the government took its most extreme form following the 1999 general elections, when the right-wing party wanted to eliminate safety contracts so as to transfer their funds to the criminal justice system, while the socialists defended the status quo. 35. Junger-Tas, 1974-1975, 702-704.

55

Evaluation_Safety.book Page 56 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

the Interior to do research on practices which had received little empirical investigation thus far. This research actually did much to improve knowledge in some fields to which access is relatively difficult (the police, in particular), because of professional confidentiality or owing to the need to obtain authorisations from agencies that were loath to collaborate in studies of their own functioning. The lack of evaluation, strictly speaking, is therefore also due to compromises between universities and the commissioning agency: evaluative research, provided it was granted some methodological freedom, provided material and findings perceived as acceptable for university-level research and teaching, as well as a basis for objective reflection for decision-making and scientific backing for the commissioning agency. This compromise worked until 1999, when reduced funding and restrictions in research themes led most academic research centres to refrain from responding to Ministry of the Interior invitations to tender, inasmuch as these no longer corresponded to the previous implicit compromise. Thereafter, academic evaluative research gradually tended to concentrate on poles of expertise increasingly confined to methodological advice, while research in criminology “inside the system” tended to gravitate toward private consultancy and auditing centres, and internal research agencies. These organisations more readily accept the requirements imposed by the commissioning agency and are viewed as more likely to provide results that are directly usable or transposable in terms of policies, but most of them are not competent in scientific evaluation of any sort, broad or narrow. To the point where research that may still be called evaluative in the broader sense is now tending toward studies and expert appraisal of internal operation, and in spite of the name, this has very little to do with any evaluative approach.

References DIRECTION GÉNÉRALE POLITIQUE DE SÉCURITÉ ET DE PRÉVENTION, 2005, Guide méthodologique pour un diagnostic local de sécurité, Bruxelles, Service public Fédéral Intérieur, Service de police Fédéral. GREENE J.C., 1994, Qualitative program evaluation, practice and promise in DENZIN N.K., LINCOLN Y. S., (Eds.), Handbook of qualitative research, London, Sage. JUNGER-TAS J., 1974-1975, Quelques réflexions sur la recherche criminologique en Belgique, Revue de droit pénal et de criminologie, 8, 702-704

56

Evaluation_Safety.book Page 57 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Belgium

LACROIX J., C. TANGE C., 1997, Évaluer les contrats de sécurité. Portefeuille de lecture, Bruxelles, Politeia. LAVAL J., 1990, Principes d’une évaluation socialement utile in Outils, pratiques, institutions pour évaluer les politiques publiques, Proceedings of the PLAN-ENA Seminar, Brussels, April-July. MINISTÈRE DE L'INTÉRIEUR, 1991, Police communale, sécurité des citoyens et recherche scientifique, Police générale du Royaume, Service Étude, conférence de presse, 19th March. PATTON M. Q., 1980, Qualitative evaluation and research methods, Newsbury Park, Sage. PONSAERS P., 2005, (Ed.), Integraal Veiligheidsbeleid 2005, Mechelen, Kluwer. PONSAERS P., JANSSEN CH., 1993, Les travaux de recherche sur la production de l’ordre et le contrôle pénal en Belgique: bilan des années 80 in VAN OUTRIVE L., ROBERT PH., (Dir.), Crime et justice en Europe. État des recherches, évaluation et recommandations, Paris, L’Harmattan. POULET I., 1995, Les nouvelles politiques de prévention. Une nouvelle forme d’action publique?, Louvain-la-Neuve, Université Catholique de Louvain, Research commissioned by Services fédéraux des Affaires scientifiques, techniques et culturelles. TANGE C., 2002, La police in MARY PH. (dir.), Le système pénal en Belgique. Bilan critique des connaissances, Brussels, Bruylant, 31-78. TANGE C., 2003, Évaluer la police? Les Cahiers de la sécurité intérieure, 53, 7-31.

Additional References CARTUYVELS Y., FRANSSEN A., HOUGARDY L., HUBERT H.-O., LEMAÎTRE A., SMEETS S., TORO F., 2003, Insécurité : un premier état des savoirs. Synthèse bibliographique relative à l’insécurité et au sentiment d’insécurité, Recherche réalisée à la demande de la Fondation Roi Baudouin, Bruxelles. CASMAN M.-TH., COSTER FR., WARTIQUE G. (dir. KELLENS G., LEMAÎTRE A.), 1992, Recherche : 4 projets communaux de prévention de la délinquance : constats après un an de fonctionnement, Recherche commanditée par le ministère de l’Intérieur, Liège, Université de Liège, Service de criminologie. DE FRAENE D., LALIEUX K., MARY PH., SMEETS S., 1997, Les contrats de sécurité et de société dans la Région de Bruxelles-Capitale, Bruxelles, Dossiers BRES, 1997, n° 34.

57

Evaluation_Safety.book Page 58 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

HACOURT G., LACROIX J., TANGE C. (Sld. POULET I.), 1999, Évaluation des contrats de sécurité et de société bruxellois. Rapport final, Recherche réalisée pour le compte du ministère de l’Intérieur et de la Fonction Publique, Police générale du Royaume, Bruxelles, Synergie, 52. HEBBERECHT P., BOVYN M., COLLE P., LIPPENS R., 1995, Politie & criminaliteit in context, Gent, RUG, Steunpunt Criminaliteit, Onderzoekgroep criminologie, Bestuurlijke Politie & Strafrechtsbedeling. KAMINSKI D., 2003, Le management de la prévention, in KAMINSKI D., GORIS P., (dir.)., Prévention et politique de sécurité arc-en-ciel, Actes de la journée d’études organisée par le Réseau interuniversitaire sur la prévention, Bruxelles, RIP, 57-72. LAUWERS S., ENHUS E., 2001, Contextualisering van de veiligheidsmonitor. Brussel: Vakgroep Criminologie, VUB. LACROIX J., TANGE C., 1997, Évaluer les contrats de sécurité. Portefeuille de lecture, Bruxelles, Politeia. MARY PH., 1998, Délinquant, délinquance et insécurité ; un demi-siècle de traitement en Belgique, Bruxelles, Bruylant. MARY PH., (dir.), 2003, Dix ans de contrats de sécurité. Evaluation et actualité, Bruxelles, Bruylant. MEIJLAERS S. (dir. WALGRAVE L.), 1992, Begeleiding en evaluatie van preventieve pilootprojecten, Leuven, Katholieke Universiteit Leuven, Faculteit Rechtsgeleerdheid, Afdeling Strafrecht, Strafvordering en Criminologie, Onderzoeksgroep jeugdcriminologie, Onderzoek in opdracht van het ministerie van Binnenlandse Zaken. PHILIPPETH K. (dir. HEBBERECHT, P.), 1993, Begeleiding en evaluatie van preventieve pilootprojecten, Gent, Rijksuniversiteit Gent, Onderzoeksgroep criminologie, Onderzoek in opdracht van het ministerie van Binnenlandse Zaken. POULET I. (dir. HOUCHON, G.), 1995, Les nouvelles politiques de prévention. Une nouvelle forme d’action publique?, Louvain-la-Neuve, Université Catholique de Louvain, Recherche commanditée par les Services fédéraux des affaires scientifiques, techniques et culturelles, Programme de recherche en sciences sociales. POULET I., BRION F., DUPONT A., 1990, Prévention en concertation au niveau local, Bruxelles, Synergie, Recherche commanditée par le ministère de l’Intérieur. SMEETS S., 2002, Les contrats de sécurité ont-ils rempli leur contrat ? Controverses et effets, in LELEUX, CL. (dir.) L’assistant social, entre aide et contrôle ?, Bruxelles, Haute Ecole Paul-Henry Spaak, IESSIID, 93-111. SMEETS S., STREBELLE C., TANGE C., 2000, Analyse des méthodes permettant de mesurer l’efficacité et la légitimité des stratégies génériques déployées par la police, Bruxelles, Université Libre de Bruxelles, Ecole des sciences 58

Evaluation_Safety.book Page 59 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in Belgium

criminologiques, Recherche commanditée par le Commandement général de la Gendarmerie, DGC/DP. STREBELLE C., 2000, Les contrats de sécurité. Évaluation des politiques de prévention en Belgique, Bruxelles, Bruylant, Coll. des Travaux de l’École des sciences criminologiques Léon Cornil. SYNERGIE, 1995, Prévention intégrée au niveau local. Manuel pratique, Bruxelles, Politeia. TANGE C., 2002, La police, in MARY PH., (dir.) Le système pénal en Belgique. Bilan critique des connaissances, Bruxelles, Bruylant, 31-78. TANGE C. (Coll. SMEETS S.), 2003, Évaluer la police de proximité ? Le cas de la Belgique, in DEBLIRE R., FERRET J., GUILLEN I LASIERRA F., RABOT A., TANGE C., VAN DER VIJVER K., ZOOMER O., LANIEL L., Recherche d’une méthodologie commune d’évaluation des réformes en matière de police de proximité en Europe, rapport de recherche pour le compte de la Commission européenne, programme OISIN 2, Paris, septembre 2003, 1, 18-198. VERSLUYS CHR. (dir. BRION F., DUPONT A.), 1993, Accompagnement et évaluation de projets-pilote communaux de prévention administrative et intégrée de la criminalité, Bruxelles, Synergie, Recherche commanditée par le ministère de l’Intérieur.

59

Evaluation_Safety.book Page 60 Tuesday, May 26, 2009 12:46 PM

Evaluation_Safety.book Page 61 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in France A Problematic Activity1 Anne Wyvekens

The concern to evaluate public policies is of relatively recent date in France. Although it has become more common over the last few years – the term evaluation is increasingly employed – “evaluation remains a problematic activity” (Lascoumes, 2006). While this statement referred to public policies in general, it is particularly applicable to the field of crime prevention and safety policies. In the strict sense of the term, they are never evaluated. There does exist however a wide range of procedures labelled as evaluative. We will first present them; and then comment on the way they are applied and to what ends2.

I.

What is meant by “evaluation”

1.

Evaluation in the strict sense of the word is not practiced in France

Evaluating a policy means gauging its effectiveness by comparing [its] results with the assigned objectives and the means employed. One could supplement this “official” broad definition3 by adding the stipulations enumerated by researchers involved in prevention and safety policies. Evaluation begins when one explores not what has been done but the consequences for the target of what has been done […] Evaluation requires being totally exterior to the programme in question. Investigating the impact […] necessitates a comparison before/after, as well as the confrontation with control groups or control areas, and finally it entails the analysis of the 1. 2.

3.

Translated by John Atherton, revised by the Author and Renée Zauberman. I thank the researchers and those actors in institutions and civil society who were generous enough to grant me interviews and thus contribute to this “inspection of the premises”. Taken from decree no. 98-1048, November 18th 1998 that deals with the evaluation of public policies

61

Evaluation_Safety.book Page 62 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

connection between the measures taken and the impact observed so as to assess whether or not the latter is due to other causes (Robert, 2005). For his study to qualify as a scientific evaluation, the researcher must first be in a position to measure not only the intervention itself (the independent variable) but also the result of the operation (the dependent variable). The result consists of the criminal symptoms that the police programme has targeted, for instance the number of car thefts or the number of fatal accidents due to speeding. These incidents must be listed in a manner that is exact, dependable and invariable (which often entails considerable difficulty). It is crucial as well to describe in detail the characteristics of the procedures set up by the police officers. It is up to the evaluator to check whether or not the operation has gone off as planned and with sufficient intensity. Exactly how did the police officers intervene? How many times? Who were their partners and did the latter play their part? When did the operation begin? (Blais, Cusson, 2007). Evaluation of this scope does not exist in France. In this domain the situation in France is often compared to that in the United States. As regards public policies we in France have had no experience with experimental or quasi-experimental evaluation: the kind of evaluation that, as in medicine for example, proceeds by comparing a test group and a control group. This lacuna reflects a difference in the scientific context; unlike the United States, where in social science research the collection of quantitative data is paramount, France has developed a more qualitative form of research, focusing in particular on process analysis influenced no doubt by the sociology of organisations (Perret, 2003). The type of evaluation that consists of comparing a “before” to an “after” is hardly any more common. In addition the socio-political context has also exercised a determining effect; in the American case, the state commands a low degree of legitimacy which can explain the need to provide proof of the effectiveness of its interventions in the public sphere; on the other hand the French government has no a priori need to justify its interventions in all those domains where the public interest is at stake (Perret, 2003). This means that for a long time there have been no (or extremely few) obligatory evaluations required, whether initiated when the experiment is first designed, or as accompaniment to its application, or as a prelude to a more extended generalisation. These remarks are particularly relevant in regard to crime prevention and safety policies. On reading through the works devoted to the evaluation of public policies one hardly ever finds examples that concern a prevention or safety policy. The Chamber of Deputies recently provided yet another eloquent illustration. The president of the Law Committee of the National Assembly, co-author of a “devastating” report on the court’s handling of 62

Evaluation_Safety.book Page 63 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in France

minor offenders4 that he had just submitted, declared in an interview: In reality we don’t have any feedback on the effectiveness of the punishments handed out. The tools for assessing the effects are sadly lacking5. In regard to urban policy, a well-placed observer recounted that a request for an evaluation of a city contract was asked of him two years after the signing: no indicator for measuring the impact had been designed at the start. How is it possible to evaluate an operation when the very conditions for an evaluation were not established from the outset?… All we had to work with were the output indicators. What is at stake then when the question of evaluation is brought up? The approach adopted by researchers differs from that of the deciders. By taking them up in succession we can introduce a degree of order into the plethora of practices, all said to be “evaluative” that are evoked on all sides.

2.

Evaluation “in the broad sense”: the researchers’ perspective

Once researchers have recognised the fact that evaluation in the strict sense of the term, that is to say quasi-experimental, does not exist, then they and the research managers tone down their stringent criteria, speaking of “evaluation in the broad sense”. They class under this heading the numerous evaluations concerned with process or functioning analysis. Many of these were focused (and are still focused) on local safety or crime prevention policies. They represent evaluation in the broad sense or, to be more exact, the first stage of a genuine evaluation that would require assessing both the manner in which a policy or a programme was implemented and its impact. Evaluation of the French Politique de la ville (urban policy) (Belorgey, 1993; Donzelot, Estèbe, 1993) is, no doubt, a good example, even if the section devoted to crime prevention was relatively minor. Local safety contracts have been the subject of this type of evaluation with varying results. The same holds true for community justice courts, and less recently, for local councils for crime prevention. In another area, that of judicial response to crime, we could cite the studies on diversion: penal mediation, restorative justice, and more recently mediatory fines. The community police programmes have been the subject of studies of a similar kind. In every instance it is a question first and foremost of scrutinising in detail the implementation of a programme, the difficulties encountered, the way it is perceived by the public, but never the “results”…. Or only in passing: In the course of research on the 4. 5.

The second section of the information mission report concerning the enforcement of criminal court decisions. Le Figaro, May 27th 2008.

63

Evaluation_Safety.book Page 64 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

functioning of the State Prosecutor's Office, whenever a prosecutor mentions the consequences of his decisions, there exists a modicum of evaluation. Then comes a very “French” invention: the Diagnostic local de sécurité, or local safety audit. A mandatory prelude to the signing of local safety contracts, these audits lay the foundations for an eventual evaluation to the extent that the contract’s "operational file cards" that lay out the measures to be adopted are supposedly elaborated on the basis of a “diagnosis” of the local situation. What we have in this case, while not the equivalent of setting up indicators, is a definition of the objectives. Here again one element of a genuine evaluation appears, but with no links to the other elements involved. In a similar vein, we have the drawing up of inventories – for France is firmly wedded to this other form of evaluation in the broad sense: the observatory. The observatory collects, collates and possibly analyses statistical data on a number of selected issues. While observatories are numerous in all domains, they are particularly frequent in the area of safety. Some are national: the OND (Observatoire National de la Délinquance/National Crime Observatory), the ONZUS (Observatoire National des Zones Urbaines Sensibles/National Observatory of Disadvantaged Urban Areas), the OFDT (Observatoire Français des Drogues et Toxicomanies/French Monitoring Centre for Drugs and Drug Addiction) – or international – the International Observatory on Violence in School; while others are local, whether it be at the municipal level (crime observatories) or the department level (observatories of truancy or school violence). Some are attached to public institutions, others set up by researchers, and yet others by economic actors, such as the transporters (the SNCF [Société Nationale des Chemins de fer Français/French National Railway Company] safety observatory and the national observatory of safety in railway and public transportation) or by social housing providers (the observatory of anti-social behaviour set up by the Social Union for Housing). But here again it is not, strictly speaking, an evaluation. In contrast to process analysis, we are here in the later stage of evaluation that deals in results. It is of interest to note that the two facets of evaluation do in fact exist… but remain for the most part disassociated; thus the Observatoire national de la délinquance has never seen fit, up to now despite the data it has amassed, to launch an evidence-based study on the police, the criminal courts, or prevention policies. Two “observatories” have, to a certain extent, proved to be exceptions to the rule: the ONZUS and the OFDT (see above). The latter has instigated research and studies that go well beyond simply collecting statistical data. In its studies, focused sometimes on drug addicts themselves, at other times on public policies – whether health or criminal – that concern 64

Evaluation_Safety.book Page 65 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in France

consumption and trafficking, the aim of providing assistance in decisionmaking is a recurring theme. L’Évaluation du plan triennal de lutte contre la drogue et de prévention des dépendances (1999-2002) (Setbon et al., 2003) consists of six volumes including one devoted to the evaluation of prevention programmes at the level of the département (Lefèbvre-Naré et al., 2003). Several monographs are explicitly evaluative (for example Setbon, 2000; Dubucq et al., 2001). These various studies deal more with process than with results, but a certain number include passages that reflect on the idea of evaluation itself. In the publications produced by the OFDT there is one paper dealing with the very notion of evaluating drug addiction interventions (Lahaye et al., 2002). As for the INSEE (Institut national de la statistique et des études économiques/ The National Institute for Statistics and Economic Studies), it provides the framework for an approach which, although it cannot claim to be evaluative in the strict sense, includes an element of comparison between a “before” and an “after”. Taking as their subject the disadvantaged urban areas, researchers compared a variety of data provided by two successive censuses (1990 and 1999). They were thus able to establish that only 150 of the 717 disadvantaged urban areas had seen any improvement in their situation during the period (Le Toqueux, Moreau, 2002; Le Toqueux, 2003). What we have here is not a study of the impact of a specific aspect of urban policy (one that would establish a link between cause and effect) but we do at least touch on the issue of the evolution of indicators that could at one point be connected to the policies adopted. In the area of criminal justice and prison – which paradoxically is a domain in which there has been little interest in evaluation in the strict sense – there exists a mass of quantitative studies. The richest “vein”, and the oldest, concerns recidivism. Inaugurated by the demographer P. Tournier, this research relied principally on cohorts studies. Applied first of all to long term sentenced convicts, these studies (for an overview of the methodological evolution see Kensey, 2007) analysed the connections between the penalty pronounced, its implementation and the percentage of recidivists (see for example Kensey, Tournier, 1994). Here again the researchers questioned the sense of the term “evaluation”. We do statistics, therefore we’re not doing evaluations. But all the same, statistical analysis, it’s an integral part of evaluation. We demonstrate for example the trends in prison population and why it evolves as it does, so it adds up to an evaluation of the policies adopted. More recently a study compared the criminal justice careers of a cohort of post-release prisoners to the criminal justice careers of a cohort condemned to non-prison sentences (Kensey, Lombard, Tournier, 2005).

65

Evaluation_Safety.book Page 66 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

3.

Evaluation as seen by institutions

The institutional approach to evaluation is different from that of researchers: more confident, and increasingly provided with examples of procedures presented as evaluative. The concern with evaluation dates from the late 1980s when Michel Rocard’s administration set out to modernise the state6. Evaluation however was to remain an internal matter, conducted by the Inspectorates of the different ministries. Yet their agents, far from eager to be outdone in their professional field, had not received any specialized training in evaluative operations. So while there was much talk of evaluation, what was done resembled an internal audit rather than an evaluation in the strict sense (Robert, 2003, 121). The evaluators were part of the institution itself and relied on its data. The first local safety contracts were evaluated in a similar manner by a joint ministerial mission directed by a government Inspector general. Mention could also be made of the evaluation of the first stages of neighbourhood social development policies (Lévy, 1988). A similarly composed mission – made up of administration inspectors and National police inspectors – was put in charge of evaluating the reform known as police de proximité, (neighbourhood policing) and in theory was to be responsible for monitoring the spreading of the programme. The mission produced four reports that were to remain secret7. The contents were later to be described as a long list of subjective opinions (Roché, 2005b). Weak in terms of methodology (no clear definition of the indicators, no control group, no care taken to adopt a standard approach in the different communities concerned…), this “evaluation” was in addition considered (as is often the case) by the reformers as more of a menace than as a tool to assist in decision-making. The reform was then generalised on the double-quick, in a time frame incompatible with the notion of a genuine experiment. The early 2000s were to mark further progress in promoting evaluations, more specifically in the area of safety but in general as well. The LOLF (Loi Organique relative aux Lois de Finance/ Constitutional Bylaw on Budget Acts) dated August 1st 2001 and applied to the administration as whole as of January 1st 2006 modified the state’s budgetary control: the traditional management of spending, ministry by ministry, was replaced by allocations earmarked for certain missions, programmes or operations. Programme objectives have to be clearly defined and refer to quantifiable indicators of 6.

7.

Michel Rocard was the French Prime Minister from 1988 to 1991, at the beginning of the second presidency of François Mitterrand (NdT). For further details, see Perret (2003). Roché (2005a) provides the details.

66

Evaluation_Safety.book Page 67 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in France

performance. While this does not make evaluation obligatory, it does at least mean that administrators are forced to argue in terms of indicators… since in theory they run the risk of budget cuts if they do not provide proof of progress. Today’s management culture favours, as a general principle, strengthening “evaluative” operations in all institutions, or at least encourages their adoption. Thus for example the September 9th 2002 law on the “orientation and programme planning for the Justice system” stipulated that the means for evaluating the educational measures and career follow-up of minors under surveillance are to be worked out in conformity with the guidelines provided by the Bylaw on budget acts of August 1st 2001… A few years later the language was to become even more peremptory: the law fixing minimum sentences for second or multiple offenders8 stipulated that an evaluation of the measures taken in application of articles 7 to 11 shall be completed no later than March 31st 2011 (art. 13). Neither were the police or the courts to be spared by the new public management techniques and performance measurement. Nicolas Sarkozy, when Minister of the Interior, inaugurated the custom of holding meetings at the Préfectures9, during which good points and bad points were allotted to the prefects of the five “best” and the five “less good” départements in regard to crime. In the Ministry of Justice, a barometer recapitulates every three months the activity of the law courts in the form of a statistical résumé (today every month for minors) (Jean, 2008).

II.

Who evaluates?

The issue of determining who is to evaluate introduces the question of the competence of the evaluators and their “neutrality”. We have already made mention of the ministerial Inspectorates. Their role is primordial even though they are both judge and party, and moreover poorly trained for evaluation operations. When it comes to professional researchers the situation is paradoxical. While they are supposed to be the best at conducting a 8. 9.

Law no. 2007-1198, dated August 10th 2007 that reinforced the fight against recidivism among both adults and minors. In France, a préfet is a high-ranking civil servant who represents the state at the level of the département or the region. Besides a range of important administrative duties, the role of the préfet is to ensure that government decisions are enforced at the local level. The term préfecture refers to the area over which the préfet has authority (in effect, the département), to the town where the administrative offices of the préfet are situated, and to these offices themselves (NdT).

67

Evaluation_Safety.book Page 68 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

genuine scientific evaluation, several interviewees let it be understood, each in his own particular manner, that France lacked “good evaluators”, a situation that could partially explain limited evaluative practice. Only a few evaluations have left an impression. If I were the General Director of the Police, I don’t see who I could turn to for an evaluation. I know ahead of time what results such-and-such a researcher will turn in. The degree with which certain methods are mastered sometimes comes up for comment – especially quantitative methods, namely opinion surveys and cohort studies; French evaluators tend to prefer qualitative methods. In addition it is often deplored that many researchers lack objectivity that their work is often biased and distorted by ideological orientations. I cannot think of one instance of the beneficial effects of a well conducted evaluation, there is always a partisan slant to the opinion. As for the researchers themselves, many of them emphasise the fact that as safety policies are on social issues, their impact is not quantifiable, and that it is artificial – even impossible – to claim that one can proceed with a quasi-experimental design. They assert in this way the pertinence of their “broad” evaluations, that is to say defined as the most comprehensive possible assistance in thinking through issues and coming to a decision. Basically though, the research milieu is in general not very enthusiastic when it comes to evaluation as if the activity itself was too operation-oriented to qualify as noble. The government’s requests for evaluations? We don’t jump at the opportunity. We don’t think of it as an essential part of our function. In one impact study they put some pressure on us to take the job, commented one research manager. Above and beyond the fact that genuine evaluation work is infrequent, it is striking to observe the extent of the gap between researchers and administrators. Researchers, on their own initiative, produce studies that serve more to stimulate scientific debate that to inform the decisions to be taken. The administrators, on the other hand, are more willing to employ their own resources (in personnel and in data), often in the name of “the need for the institutional actors to appropriate the evaluation as their own”. The organisations that fund research play an interface role, among them the research mission Droit et Justice (Law and Justice)10 for the court and 10. A public interest grouping. The founding members were the Ministry of Justice (50% of the membership), the Centre national de la recherche scientifique (30%), the École nationale de la magistrature (National Magistrates’ School), the Conseil national des barreaux (National Lawyers’ Council), the Conseil supérieur du notariat (Notaries’ High Council (5%). Associate members include the Ministry of Research, the Institut des hautes études sur la justice (Institute of Advanced Justice Studies), and the Association française pour l'histoire de la justice (French Association for the History of Justice).

68

Evaluation_Safety.book Page 69 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in France

prison system, and the Institut National des hautes études de sécurité (INHES, National Institute for Advanced Safety Studies)11 for police affairs. Each of them – operating within different frameworks and with different protocols – elicit, finance, and evaluate research. Once again, genuine evaluation plays a minor role. Only in the latter organisation there was a call for an intermediary status, located somewhere between third-party neutrality and institutional control, for evaluating the “neighbourhood policing” reform. The “immigrants” (Ocqueteau, 2003) were researchers temporarily assigned a post of research director in the institution, and in this capacity they accompanied the in-house evaluation operations in the field, thus providing, at least partially, the element of exteriority that was lacking. In the area of local safety policies, the evaluations or audits conducted on site were often the work of a third category of actors, the consulting firms. Today, on the national level, it is also these private firms that land the big contracts – from the Délégation interministérielle à la Ville (Interministerial Urban Delegation) for instance – or who are preferred by Directorates, as is the case for the prison administration that appreciates “results that are clear, concise with blueprints”. In addition to the brevity of their conclusions, it is the rapidity with which these firms work that makes the difference. The issue of knowing who is evaluating comes down to knowing who is financing the evaluation (still in the broad sense of the term). France is provided with a national research institute, the CNRS (Centre National de la Recherche Scientifique) staffed with full-time research personnel classed as public service employees. Thus much research work can be accomplished without outside financing. But funds have to be raised – through contracts with the government departments or private firms – in order to cover the costs such as that of field work and pay the salaries of research personnel and (young) non-statutory researchers. The independence that results from this set-up has been reduced by a tendency, that is on the rise, for financing to be funnelled not through the research units themselves but through an agency in charge of spending, the ANR (Agence Nationale pour la Recherche/National Agency for Research), which funds research through competitive calls to tender. The growth of a market in expertise concerning safety programmes and in particular concerning the audits on which they must be based has led to questioning as to the reliability of these audits. Cost concerns have led to the practice of using stereotyped audits, applying the same interpretative grid to very different local contexts – a far cry from the audits “shared” 11. A public institution for training and research attached to the Ministry of the Interior.

69

Evaluation_Safety.book Page 70 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

between local actors as is the case for the most motivated municipalities (see in particular Donzelot, Wyvekens, 2004) – a development that has earned those who produce the formulaic audits the somewhat derogatory name of “pseudo-research consultants” (Robert, 2003, 125).

III. What purpose does “evaluation” serve? The possible uses to which “evaluation can be put are many. Process evaluations and a posteriori descriptions of interventions – particularly in the case of “shared” operations that have been “appropriated” by the actors directly concerned – provide opportunities for improving cooperation and methodology, for relaunching the measures or programme in question. This is the purpose of the collaborative services in matters of audits or safety policies offered cities by the French Forum for Urban Safety; the Forum committee that oversees these projects aims to be as open as possible to the various actors concerned. The remoteness of those who perform the evaluation from those who commission it – a subject already evoked – is here more or less intentionally maintained, proof in itself that this type of procedure is relatively infrequent. The follow-up of an experiment is another possible use. It was said to be the case for the “neighbourhood policing” reform. But although the trappings were there, reality was another matter since the generalisation of the reform went ahead at a rapid pace without taking into account the reports that were appearing simultaneously. When it was decided to institute an electronic monitoring programme the procedure adopted was largely the same (Kensey et al., 2003). The study that the prison administration Directorate had entrusted to a CNRS research centre was never made use of, even though research indicated a number of defects in the way the programme was being implemented. It would be unfair to draw a general conclusion from these selected examples, but it is none the less true that they illustrate, each in its own way, the “problematic” nature of evaluation in France. On a more general level, evaluation, even in the broad sense, is by nature likely to nourish robust discussion and political debate – and go as far as requesting that all actors “be held accountable” whether it be to funders, to those who benefit from interventions, or to the citizenry as a whole. But the fact is that the results of research, when they do exist, are rarely made public. Much of the work conducted by the government or on its initiative is for purely internal use, and not for distribution (or for restricted distribution only). Take a recent example: the Ministry of Education requested that the study of CCTV in the Île-de-France lycées (high schools) 70

Evaluation_Safety.book Page 71 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in France

(Le Goff et al., 2007, see below) not be made public. One wonders as well what use was made by the Ministry of the Interior of an evaluation of CCTV realised (and financed) by the INHES: several months after it was submitted the report had neither been published nor its conclusions disseminated… despite the fact that CCTV has now spread exponentially into all fields at the instigation of this same Ministry. Other major public “debates” on security issues – minimal sentences, post-term safety detention12 – continue without referring to any evaluative studies. During the thirty years they [evaluation studies] have existed, they have never had any effect on political decisions or democratic debate. Political decisions in the area of safety appear in France as particularly disconnected from the minimal evaluation that does exist and on a more general level disconnected from any research work on the policies in question. This opinion, which is shared by countries more committed to evaluation, reflects the eminently political nature of this type of governmental intervention in France. As one of our interlocutors put it, the “depoliticisation” often results in enhanced knowledge of an issue. Thus drug addiction is the subject of a “real effort” to acquire knowledge, in particular on the initiative of the French Monitoring centre for Drugs and Drug Addiction. As for the performance assessments realised in application of the LOLF, or more generally within the ministries themselves, they belong more under the heading of “management control” than that of genuine evaluation.

By way of illustration… We will conclude by citing two recent studies which, each in its own fashion, can be seen to be auspicious for the future of evaluation in France. While they do not constitute “genuine” evaluations, they both open up promising avenues.

1.

CCTV in the lycées of Île-de-France. Uses and impact

This research, the work of the Institut d'aménagement et d'urbanisme d'Îlede-France (the Paris Region Institute of Urban Planning) (Le Goff et al., 12. A law recently incorporated in the French criminal code (February 25th 2008) provides that persons convicted for any of a number of very high profile crimes and still regarded as dangerous after having served their prison sentence can be kept in a socio-medical detention facility until declared harmless.

71

Evaluation_Safety.book Page 72 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

2007), is exceptional, first of all in its origin. It was initiated at the request of the “Green” (ecological) party representatives for the Île-de-France region who made of it a condition for voting the budget for CCTV in the lycées13. It was thus explicitly specified – and the case is sufficiently rare to be worthy of note – that a political decision was to be based on the knowledge acquired of the issue. The study employed a qualitative methodology (interviews, observation) which provided information on the uses – which were varied – of the scheme, the means to assess its impact on criminal activity, the unintended results, as well as the discrepancies between the actual effects and those which both the personnel and the elected representatives had imagined. The study revealed that the efficiency was significantly more limited than the personnel had expected. But also it served objectives quite different from those foreseen by the representatives: they had seen in it only a means for guarding against intrusions, but they discovered uses connected with the maintenance of discipline within the establishment.

2.

The National Observatory of Disadvantaged Urban Areas

The August1st 2003 law for “city planning and urban renovation” provided for the development and application of programmes destined for depressed areas, involving employment, housing, health, schooling, safety and law and order. It created an Observatoire national des zones urbaines sensibles (ONZUS, National Observatory for Disadvantaged Urban Areas) assigned the responsibility of plotting the evolution of social inequality and differences in economic development in each of the deprived urban zones, of monitoring the implementation of public policies designed to benefit them, assessing the specific measures taken and evaluating their effects by comparison with the objectives and in terms of the impact indicators listed [in the appendix]. In France a set-up of this kind was tantamount to a revolution. A link was to be established between results (in the case at hand: the percentage of cases solved, the number of victims, the fear of crime/sense of insecurity) and the resources engaged (the number of local safety contracts, the number of social mediation agents, etc.)14. While not based on a random experimental design, this evaluative scheme collects and centralises indicators. The ONZUS worked in close collaboration with the OND which furnished statistics on the offences committed15, and the INSEE that provided data on the fear of crime and sense of insecurity in the 13. In France it is the regional councils that finance lycées facilities.

72

Evaluation_Safety.book Page 73 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in France

neighbourhood. Yet while this represents a step forward towards evaluation, it is regrettable that the corresponding financial means have not been allocated (Roché, 2005): the evaluation budget is not calculated as a percentage of the total subsidy granted. Moreover the observatory does not possess sufficiently detailed information on the policies adopted, which makes it difficult to judge their results. The incidents that are recorded in the neighbourhoods concerned are compared with figures drawn from the entire public safety district. Links are established between the inhabitants’ sense of insecurity and the varying local characteristics. And above all, with this observatory constituting a permanent structure, data can be collected in an unvarying format which is indispensable for more elaborate analyses which it is hoped will be conducted in the future (ONZUS, 2005). Even if the quasi-experimental type of evaluation does not exist in France, there is no lack of studies with evaluative significance or evaluative presumptions. But the problem that gives most food for thought is of a different nature: it is the absence or shortage of a “culture of evaluation”. Lack of concern to be held accountable, to “accept to be seen”, “to take steps to be seen”. The question at issue is less that of a choice of method – 14. 6.2.1 Indicators for results: - number of crimes and misdemeanours (committed in the disadvantaged urban areas) as recorded by the police and the gendarmerie classed by category (national police crime record “4001”): intentional bodily harm both criminal and felonious except those resulting in death, unarmed violent robbery of women in public areas, destroying and vandalising private vehicles, residential burglary, destroying and vandalising public property, trafficking and reselling drugs, mistreatment and abandonment of children); - percentage of above crimes solved; - number of case of contempt or assault directed at police agents (police crime record “4001”); - number of incidents recorded in the schools based on the school violence data base; - utilisation of the annual INSEE survey (survey of household living conditions, questions concerning the sense of security). 6.2.2 Indicators of means: - number of special unit agents (juvenile squads and prevention of juvenile crime squads) assigned to the district in which a ZUS is situated; - number of victim support centres located in the communes in which a ZUS is situated; - number of sites providing access to the law (law and justice centres, legal information centres); - number of local safety and crime prevention contracts; - number of specialised crime prevention social workers; - number of social mediation agents. 15. 28 of the 107 indexes included in the national police crime record “4001” account for 70 to 80 % of the offences.

73

Evaluation_Safety.book Page 74 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

a subject amply debated in the course of this seminar – but a question of the relationship between science and government.

References BELORGEY J.M., 1993, Évaluer la politique de la ville ; rapport du Comité d’évaluation de la politique de la ville, Paris, Conseil national des Villes. BLAIS E., CUSSON M., 2007, Les évaluations de l'efficacité des interventions policières: résultats des synthèses systématiques, in CUSSON M., DUPONT B., LEMIEUX F., (Dir.), Traité de sécurité intérieure, Montréal, Ed. Hurtebise, 115-129. DONZELOT J., ESTÈBE PH., 1993, L’État-animateur, Paris, Esprit. DONZELOT J., WYVEKENS A., 2004, La magistrature sociale. Enquêtes sur les politiques locales de sécurité, Paris, La Documentation française. DUBURCQ A., SORIA J., AHOUANTO M., PÉCHEVIS M., 2001, Évaluation du dispositif 'Familles d’accueil pour toxicomanes', Paris, OFDT. JEAN J.-P., 2008, Le système pénal, Paris, La Découverte, coll. Repères. KENSEY A., 2007, Prison et récidive, Paris, Armand Colin. KENSEY A., LOMBARD F., TOURNIER P.V., 2006, Sanctions alternatives à l’emprisonnement et 'récidive'. Observation suivie, sur 5 ans, de détenus condamnés en matière correctionnelle libérés, et de condamnés à des sanctions non carcérales (département du Nord), Paris, ministère de la Justice, direction de l’Administration pénitentiaire, coll. Travaux et documents, n° 70. KENSEY A., PITOUN A., LÉVY R., TOURNIER P., 2003, Sous surveillance électronique. La mise en place du 'bracelet électronique' en France (octobre 2000-mai 2001), Paris-Guyancourt, DAP-PMJ1-CESDIP, coll. Travaux et Documents – Études et Données pénales. KENSEY A., TOURNIER P., 1994, Libération sans retour, devenir judiciaire d’une cohorte de sortants de prison condamnés à une peine de trois ans ou plus, Paris/Guyancourt, SCERI, Travaux et documents, n° 47, CESDIP, Études et données pénales, n° 69, [http://www.cesdip.fr/IMG/pdf/ EDP_69.pdf]. LAHAYE,D., PADIEU R., RICAUX,G., 2002, Évaluer l’intervention en toxicomanie : significations et conditions, Paris, OFDT/CAST, [http:// www.ofdt.fr/BDD/publications/docs/eval_intervention.pdf]. LASCOUMES P., 2006, L’impossible évaluation des actions publiques ? in LASCOUMES P., VARONE F. (dir.), L’évaluation des politiques publiques entre enjeu politique et enjeu de méthode, Les Cahiers du CEVIPOF, 44, 7-24.

74

Evaluation_Safety.book Page 75 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in France

LEFEBVRE-NARÉ F., DEVAUGERME F., LIORET C., 2003, Évaluation des programmes départementaux de prévention, Paris, La documentation française, [http://lesrapports.ladocumentationfrancaise.fr/BRP/ 044000624/0000.pdf]. LE GOFF T., LOUDIER-MALOUYRES C., LAVOCAT CH., DAUTHEVILLE M., 2007, La vidéosurveillance dans les lycées en Île-de-France. Usages et impacts, Paris, IAURIF, [http://www.iaurif.org/fileadmin/Etudes/etude_221/ nr_437_la_videosurveillance.pdf]. LE TOQUEUX J.-L., 2003, Lieux de résidence et sentiment d’insécurité, in PUMAIN D., MATTEI M.-F. (dir.), Données urbaines, 4, Paris, Anthropos, 147-154. LE TOQUEUX J.-L., MOREAU J., 2002, Les zones urbaines sensibles, INSEE Première, mars, 835, [http://www.insee.fr/fr/ffc/docs_ffc/IP835.pdf]. LÉVY F., 1988, Bilan-perspectives des contrats de développement social des quartiers, Commissariat général au Plan, Paris, La Documentation française. OCQUETEAU F., 2003, Comment évaluer l’impact du travail des policiers de proximité ? Criminologie, 36, 1, 121-141. OBSERVATOIRE NATIONAL DES ZONES URBAINES SENSIBLES, 2005, Rapport 2005, Paris, Éditions de la DIV, [http://www.ville.gouv.fr/ article.php3?id_article=145]. PERRET B., 2003, L’évaluation des politiques publiques. Problématique générale, Informations sociales, 110, 14-25. ROBERT PH., 2003, The Evaluation of Prevention Policies, European Journal of Crime, Criminal Law and Criminal Justice, 11, 1, 114-130. ROBERT PH., 2005, La sociologie du crime, Paris, La Découverte, coll. Repères. ROCHÉ S., 2005a, Police de proximité. Nos politiques de sécurité, Paris, Seuil. ROCHÉ S., 2005b, The Transferability of Evaluation and the "What Works" Approach in France, European Journal on Criminal Policy and Research, 11, 3-4, 297-320. SETBON M., 2000, L’injonction thérapeutique: l’évaluation du dispositif de prise en charge sanitaire des usagers de drogue interpellés, Paris, OFDT, étude n° 21, [http://www.ofdt.fr/BDD/publications/docs/it.pdf]. SETBON M., GUÉRIN O., KARSENTY S., KOPP P. (dir.), 2003, Évaluation du plan triennal de lutte contre la drogue et de prévention des dépendances (1999-2002), Paris, OFDT.

75

Evaluation_Safety.book Page 76 Tuesday, May 26, 2009 12:46 PM

Evaluation_Safety.book Page 77 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in the Netherlands1 Karin Wittebrood

Increasing social safety – i.e. reducing actual crime and nuisance as well as improving perceived or subjective safety – has been high on the government list of policy priorities in the Netherlands since the launch in 2002 of the Safety Programme (Naar een veiliger samenleving – Towards a safer society) by the first government under Prime Minister Jan Peter Balkenende. Attention in this Programme focuses mainly on those forms of crime and nuisance which affect citizens and companies/institutions in the public space. The successive Balkenende governments have targeted a national reduction in crime and nuisance of 25% between 2006 and 2010 (compared with 2002). Perceived safety must also increase substantially in this period. When the fourth Balkenende government took office in early 2007, the general impression was that a great deal had been set in motion and achieved with the 2002 Safety Programme: the Netherlands has become safer and crimes against citizens and businesses have fallen. The project ‘Safety starts with Prevention’ (Veiligheid begint bij Voorkomen) has recently replaced the 2002 Safety Programme. The Balkenende IV government has however indicated that the objectives formulated in that Safety Programme will remain in force and that where necessary measures introduced will be continued. The optimism voiced about the effectiveness of the Safety Programme was not universally shared. In 2006 the Netherlands Court of Audit stressed that it is virtually impossible to determine what contribution central government policy makes to resolving the social issues for which that policy is deployed and that there is no solid substantiation of the relationship between resources, measures and desired effects (TK 2005/2006, 6). Partly based on the study by the Netherlands Court of Audit and the recommendations derived from it, the Minister of the Interior and Kingdom Relations and the Minister of Justice, supported by the Minister of Finance, decided to initiate a research programme to investigate the ‘social costs of safety’ (Maatschappelijke Kosten van Veiligheid). On the one 1.

Based on van Noije, Wittebrood (2008).

77

Evaluation_Safety.book Page 78 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

hand, this research is intended to meet the criticism by the Court of Audit that there is no theory underpinning the policy, that the policy is insufficiently substantiated and that there is insufficient information on the amount of money spent on safety. On the other hand, the ministers concerned wish to use the research findings to obtain a better insight into the returns on investments in safety, thus providing a more solid basis for future policy choices. In this paper we will answer three questions based on an analysis of the presumed and actual effects of the Dutch safety policy carried out by the Netherlands Institute for Social Research (Van Noije, Wittebrood, 2008). 앫 Which assumptions underlie social safety policy? 앫 To what extent and in what way are the measures taken effective? 앫 How plausible are the assumptions which underlie social safety policy?

I.

Which assumptions underlie social safety policy?

The government uses a variety of measures to increase social safety, varying from more police on the streets and the use of camera surveillance to providing care and support for at-risk young people and imposing harsher penalties. Preventive effects are expected from all these measures. In other words, it is assumed that implementing these measures will reduce crime and nuisance in the future. It is also expected that these same measures will cause citizens to feel safer. The measures taken can be categorised within a number of strategies intended to increase social safety. In this study we distinguish three main categories: (1) law enforcement; 2) developmental prevention; and (3) situational prevention. In addition, we devote attention to systemic measures, which occupy an important place within the Safety Programme.

1.

Law enforcement

The most important general assumption underlying law enforcement is that it has both a specific and a general preventive effect and that this contributes to the restoration of the legal order. The Safety Programme frequently refers to intensification within the mainstream law enforcement system. There are two central target groups in the Safety Programme: frequent offenders and young people. However, some of the assumptions also apply to the offender population as a whole. In the first place, the Programme aims to increase the chance of offenders being caught, arguing that too many non-interventions undermine the deterrent effect of 78

Evaluation_Safety.book Page 79 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in the Netherlands

sanctions, reduce the effectiveness of sanctions imposed, and dent the credibility of the law enforcement system. In the second place, there is an assumption that shortening the throughput times within the judicial system and therefore being able to impose penalties more quickly will increase the effectiveness of the sanctions imposed, thus reducing recidivism. In the third place there is the assumption that ‘customisation’ in the application of sanctions reduces the chance of recidivism: by opting for person-specific rather than offence-specific sanctions, it is argued, both the sanctions and the aftercare will be made more effective. If the sentence is not just based on the most recent offence, but on the whole criminal dossier offender can be longer detained. The extra time can be used for more intense, hence effective, rehabilitation.

2.

Developmental prevention

In the second strategy, the general assumption is that individual offending behaviour is determined by a combination of factors including life situation, peer pressure, the quality of parenting and future prospects. According to this view, the success of interventions depends partly on how early those interventions take place. Consequently, measures in this strategy are targeted mainly at young people. The Safety Programme cites risk factors for youth crime (such as premature school dropout, poor command of the Dutch language, unemployment and alcohol and drug abuse). It also stresses the need to ensure positive development by young people, for example by providing help and support. The details and implementation are however largely left to the Ministry of Health, Welfare and Sport (VWS), the Ministry of Education, Culture and Science (OC&W) and the Ministry of Social Affairs and Employment (SZW) in the case of young people who have not (yet) come into contact with the police. Some interventions, especially those involving the provision of assistance, can also form part of a sanction and therefore fall within the law enforcement strategy. The Safety Program presents no assumptions about the effectiveness of measures either promoting protective factors or fighting risk factors.

3.

Situational prevention

The strategy of situational prevention is based on the idea that modifying the ‘opportunity structure’ – for example by making it more difficult or more unattractive for potential perpetrators – will prevent offences being committed. The Safety Programme focuses on specific elements of the 79

Evaluation_Safety.book Page 80 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

strategy of situational prevention. An important assumption is that repeated breaking of the rules combined with a lack of visible supervision in the public domain provides a breeding ground for an enforcement deficit that is widely felt throughout the community. The intention is therefore to send out a signal by consistently enforcing the ‘low-level norm’. Another key assumption is that extra supervision at ‘hot spots’ and ‘hot times’ produces the greatest returns. It is also assumed in the Safety Programme that, among other things by taking physical security measures, potential perpetrators can be discouraged from committing offences.

4.

Systemic measures

In order to improve the functioning of the measures taken within the above three strategies, the Safety Programme also devotes attention to systemic measures. The most important of these, the chain approach, is mainly concerned with strengthening the relationships between different partners in the chain. In the area of law enforcement this means in the first place liaison between partners within the judicial chain: in 2006 the police were expected to deliver 40,000 extra suspects to the Public Prosecution Service, something which only makes sense if the Public Prosecution Service is able to respond adequately by adapting its working processes, while the courts must also not be overwhelmed by the number of cases being presented. The chain approach also embraces the connection between the judicial process through which the perpetrator passes and the subsequent social support and rehabilitation they receive from the probation and other services. In the developmental prevention strategy, the chain approach is concerned more with ensuring a seamless connection between the different care organisations, schools, local authorities and the police, so that problems can be detected at an early stage and at-risk young people are not lost from view or harmed due to poor communication or contradictory strategies. The core focus in the situational prevention strategy is public-private partnership. Systemic measures are used to make substantive measures more effective by creating appropriate conditions and improving working processes. The success of these measures depends on how they are implemented.

80

Evaluation_Safety.book Page 81 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in the Netherlands

II.

To what extent and in what way are the measures taken effective?

In order to substantiate safety policy it is necessary to know whether the measures deployed have had the envisaged impact on social safety. Evaluation research should provide an answer to this. A key milestone in criminological effect evaluation research was the Sherman report, published in 1997, entitled Preventing crime. What works, what doesn’t, what’s promising (Sherman et al., 1997). This report was updated, expanded and improved in 2002 and 2006 under the title Evidence-based crime prevention (Sherman et al., 2002, 2006). This report summarises 675 scientific evaluations of measures taken in various countries aimed at reducing crime. In addition, an international network was set up some years ago to encourage effect evaluation research: the Campbell Collaboration Crime and Justice Group (Farrington, Petrosino, 2001). This group stresses the importance of systematic summaries of effect evaluations based on a (quasi-)experimental study design. Updated summaries are published regularly (see www.campbellcollaboration.org). In our study we followed the same working methods as Sherman et al. (2002) and the Campbell Collaboration Crime and Justice Group. In the first place, this means that we assessed the effect evaluations found mainly on the basis of their internal validity. To do this we used the (Maryland) Scientific Methods Scale (SMS), a five-point scale which shows simply and clearly the differences in methodological quality between effect evaluations (Farrington et al., 2002). Based on a systematic literature survey, we mapped out the empirical findings of Dutch effect evaluations and supplemented them with findings from other countries. We concentrated on evaluation research which is able to demonstrate objectively that social safety has changed as a result of a given measure. This means that the chosen outcome measure (generally relating to recidivism, criminality, nuisance or subjective (lack of) safety) must have been measured explicitly. No restrictions were applied during the literature survey regarding the quality of the effect evaluations; this enabled the quality of the Dutch studies to be assessed as well. In addition, no restrictions were applied concerning the year in which the evaluation was carried out; the main reason for this was that a number of good effect evaluations were carried out in the 1970s and 1980s, which we wanted to include in this study because they provide an insight into the plausibility of certain assumptions. The search criteria produced a list of 149 Dutch effect evaluations that have been carried out in recent decades (see table 1). A content analysis was

81

Evaluation_Safety.book Page 82 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

then performed on each study, and at the same time a quality assessment was carried out. We assessed the internal validity explicitly: the higher the assessment score, the greater the likelihood that the measure in question has actually brought about the desired change. As a minimum, a pre-study and post-study measurement must have been carried out and there must be an intervention group or area and a control group or area. This is important, because in order to be able to make any statements about the potential effects of measures on the envisaged objective, it is necessary to know whether a change has taken place and whether it would also have taken place without the intervention. Of the effect evaluations found, 69 had at least a quasi-experimental design. If certain characteristics had moreover been subjected to statistical analysis or if the intervention had been assigned arbitrarily to the experimental group and the control group, these statements can be made with even more certainty. Generally, it was found that the majority of the measures deployed by the government in recent decades had not been evaluated to determine their effects on social safety. A sizeable proportion of the measures implemented moreover failed to meet the minimal study design requirement (i.e. a quasiexperimental design) for an effect evaluation. As a result, it remains unclear in these cases whether any improvement in social safety is due to the measure in question or to something else. In addition, measures were often evaluated within a specific period. For example, research on the effects of detention regimes took place mainly in the 1970s, while research on training orders for young people below the age of criminal responsibility is concentrated in the 1990s and research on the effects of camera surveillance mainly in the present century. This concentration of attention corresponds reasonably well with developments in social safety policy. This is hardly surprising, because many effect evaluations take place at the request of government ministries or local authorities. The quality of the effect evaluations depends on who performed them. Evaluations carried out by researchers attached to the organisation which implements the measure are of the lowest relative quality; virtually none of these internal evaluations allow any statements to be made about the effectiveness of the measure studied. The quality of effect evaluations carried out by commercial research agencies varies widely: some are good, some less so. One characteristic shared by all these evaluations, however, is that – regardless of their internal validity – they are often less critical when describing the outcomes than evaluations carried out by (para-)university institutions and independent government research agencies. The evaluations carried out by these bodies are of better quality on average and are also more critical than evaluations performed by others. 82

Evaluation_Safety.book Page 83 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in the Netherlands

Total

Law Developmental Situational enforcement prevention prevention 54 35 63

Internal validity (SMS) 1 2a 2b 3 4 5

1 9 7 16 21 1

1 12 4 11 5 2

2 27 21 8 5 0

Year of publication < 1986 1986 – 1990 1991 – 1995 1996 – 2000 2001 – 2008

7 3 14 12 18

0 4 6 4 21

3 10 10 10 30

Data sources used Survey/interviews Registrations Both

6 36 12

22 8 5

15 8 40

33

4

10

9 6 6

15 4 12

12 11 30

Research institute Independent government University Commercial Internal

Table 1: Number of studies within different strategies

83

Evaluation_Safety.book Page 84 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

III. How plausible are the assumptions which underlie social safety policy? By confronting the policy assumptions made in the Safety Programme with the available knowledge on the effectiveness of specific measures in increasing social safety, we obtain an insight into the plausibility of the assumptions made and are therefore able to answer the third research question. We again do this separately for each strategy, concentrating on the assumptions which we can classify as plausible or implausible and on major omissions in the policy theories.

1.

Law enforcement

In the first phase of law enforcement – investigation – the main policy assumption is that sure (or consistent) punishment, via different routes, will lead to an increase in social safety. Many measures are accordingly aimed at increasing the chance of being caught. Based on our study, we can conclude that increasing the chance of arrest has a positive impact mainly where the police concentrate their efforts on hot spots and at hot times and therefore take suspects ‘out of circulation’, at least temporarily. A high chance of being caught also appears to have a general deterrent effect. This general preventive effect does not apply where the police arrest a suspect after the event (reactive arrest), but only operates in the case of specific and anticipatory action. However, there is little plausibility in the assumption that an increased risk of being caught has a deterrent effect on suspects themselves. If anything, it actually lowers the threshold to committing new offences, especially among minors and perpetrators of relatively minor offences. In the prosecution phase, the policy is to reduce the case throughput time, so that suspects can be convicted and punished more quickly. It is assumed that this increases the effectiveness of the sanctions imposed. Dutch research in this area shows that this assumption holds little water: more rapid imposition of punishments has not been demonstrated to have a positive effect in reducing recidivism. A central notion in the sanctioning and aftercare phase is that a personspecific approach enables more ‘customised’ sanctions to be imposed, thus increasing the effectiveness of punishments. In the first place, a great deal is expected of custodial sentences. Imprisonment is found to be effective for the duration of the detention: suspects and convicted offenders are after all temporarily rendered harmless and have (virtually) no opportunity to commit offences. As assumed in the policy theory, however, it is not 84

Evaluation_Safety.book Page 85 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in the Netherlands

plausible to assume that incarceration will contribute to reducing recidivism after release. In fact there are even indications that custodial sentences may have a negative impact in this regard. The policy takes no account of this possibility at all, whereas the consequence for the longer term may be that people are being released who are at increased risk of recidivism. This means that detention combined with treatment, as well as other kinds of punishment, deserve attention as alternatives to custodial sentences. In line with the policy assumption, short custodial sentences in any event appear useless. If a prison sentence is chosen, it is better for it to be of sufficient length to enable the offender to pass through a resocialisation programme, provided such a programme is available in practice. Although the Safety Programme looks at treatment programmes for prisoners, it largely ignores assumptions on an effective approach to these programmes. This is another gap in the policy theory. Research shows that cognitive behavioural therapy and social skills training are the most effective. The policy assumption that phased reintegration and aftercare based on supervision and support contributes to reducing repeat offending can be described as plausible. Resocialisation programmes and support are also important for detainees with conditional sentences. The policy assumption that using these punishments as an alternative to imprisonment increases the sanctioning capacity is plausible in the short term. In the longer term this assumption is only valid if these punishments are effective in reducing recidivism. This appears to be the case for conditional prison sentences. Where the punishments merely restrict freedom of movement and there is no resocialisation or support, the risk of recidivism is considerable. This means that conviction at a later moment will absorb sanction capacity at that time. In the sanctioning and aftercare phase, recurrent offenders – often drug addicts – are an important target group. They are regarded as the main cause of both frequent crime and the deficiencies in law enforcement. Longer prison sentences (up to two years) are a particularly important measure for this group. From the perspective of protecting society, this measure is effective. For a select group – those who are assessed as the most open to influence – this time is used for behavioural interventions. There are indications that this reduces recidivism after completion of the programme by this group, but the rest can expect a more stringent regime, and if this means that prisoners spend most of their time in their cells this is likely to have a detrimental effect on recidivism. The second target group identified in the sanctioning and aftercare phase in the Safety Programme are young people. For young people who commit 85

Evaluation_Safety.book Page 86 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

less serious offences, the policy assumption that providing parenting support has a positive influence in reducing recidivism is empirically supported, especially if the support is supplemented by other forms of help. The idea of referring these young people to the Halt procedure is also supported. These and other ‘alternative punishments’ in any event appear to be much more effective than options such as the American ‘scared straight’ programmes, which can even serve to increase recidivism. Young people who display serious and frequent criminal behaviour are often placed under supervision and treatment or even placed in a re-education facility. As with adults, cognitive behavioural therapy and social skills training appear to offer the most promising results for young people. The policy assumption that re-education focusing on discipline (such as the Glen Mills School and the approach used in Den Engh) helps to reduce recidivism is therefore not very plausible. The criticism of the disciplinebased re-education methods can as yet not be extrapolated to the zerotolerance policy, because virtually nothing is known about the effectiveness.

2.

Developmental prevention

We can be brief on the plausibility of the policy theory which places the emphasis on juvenile interventions. This theory does not describe any mechanisms which link means and ends, but merely links problems (delinquent behaviour) to causes (such as school dropout, parenting quality and alcohol use). We are therefore not able to assess the plausibility of this approach. Broadly, the factors cited in the policy theory are plausible as important risk factors for delinquent and aggressive behaviour, although the causal relationship that is assumed in the policy theory remains open to question. It would be desirable for future policy to set out the assumed relationship between the measures and the risk factors; only then can an assessment be made as to whether the policy has a chance of succeeding.

3.

Situational prevention

The strategy of situational prevention focuses primarily on measures which are clearly visible and which are intended to send out a normative signal. This strategy is exclusively concerned with general prevention; as soon as someone is actually caught, we are in the domain of law enforcement. The policy assumption that formal supervision (i.e. the police) leads to a reduction in crime and nuisance can be regarded as plausible, especially where it is concentrated on hot spots and hot times. 86

Evaluation_Safety.book Page 87 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in the Netherlands

There is also support for the idea that a visible police presence is effective in increasing subjective safety. Where formal supervision is carried out using camera surveillance, the observations on effectiveness are unclear. Camera surveillance is of most use in facilitating rapid police intervention and detection/investigation. The policy assumption that camera surveillance, as a form of formal supervision, prevents crime and nuisance, is doubtful. The assumption that camera surveillance can increase public perceptions of safety can be rejected more firmly. Measures designed to improve the protection of potential targets of crime and thus make it more difficult for potential perpetrators to commit offences, also receive a good deal of attention in the Safety Programme. These measures are concerned mainly with technical security (an area where members of the public and businesses are encouraged to take their own responsibility). Predominantly positive research results mean that this policy assumption on protective measures can be regarded as plausible. Little is known about the effect of such measures on perceived (lack of) safety. All in all, the measures to reduce opportunity which are emphasised in the Safety Programme and which the government regards as its own responsibility, rather than that of the private individual, are heavily focused on the perpetrator. The idea of the strategy of situational prevention is however more wide-ranging. It appears that when this policy is implemented at local level, the full range of measures is deployed and measures are (also) taken which target potential victims and situations. In the policy theory, however, a choice has been made in favour of measures which are closely related to those used in the law enforcement strategy. This is a pity, because measures such as functional supervision (e.g. by wardens and inspectors) appears promising. Although the research results are by no means uniform, the findings do point in a positive direction as regards reducing frequently occurring crime and enhancing perceived safety.

4.

General conclusion about Safety Programme

How much do we know about whether the Safety Programme has contributed to the recent fall in crime and the improvement in perceived safety? The diverse nature of the Safety Programme means it is not possible to express a definitive opinion on the Programme as a whole. Some parts of it offer promise in reducing crime and nuisance (such as the deployment of the police at hot spots and hot times, attention for parenting support and the use of quality marks), while others do not (e.g. harsher detention

87

Evaluation_Safety.book Page 88 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

regimes, arrests for minor offences and disciplinarian re-education facilities), while the effect of others is unknown (e.g. camera surveillance). The tackling of frequent offenders which lies at the heart of the Safety Programme illustrates the mixed results achieved with the Programme. For example, the picture on reducing ‘revolving-door crime’ is a positive one; the police and justice system not only ensure that frequent offenders are removed from the streets, but also that they are not immediately back on the streets the next day to simply carry on their criminal activities. Here we see that the policy is achieving good results in the short term, with immediate protection for society. This may also have a positive impact in terms of social redress. Our conclusion on the target group and the prosecution process through which arrested frequent offenders pass is a much more critical one. In the first place, there is a greater readiness to arrest suspects for minor infringements, an approach which can have the unintended effect of encouraging a criminal career. Second, longer custodial sentences are imposed in order to combat revolving-door crime, but without any clarity regarding further behavioural interventions or how the perpetrator spends their time during and after their period of detention. In the area of reducing recidivism, which is key to success in the longer term, there are thus no indications that ‘sure, faster and more severe punishments’ achieve greater success than would have been the case with a different policy. * Our conclusion is that it is often not (yet) possible to state whether or not policy assumptions are plausible, partly because in many cases we do not know whether these measures contribute to increasing social safety. These findings lead to the observation that the quality of evaluation research needs to be greatly improved in order to provide a better substantiation for safety policy. It is very important that effect evaluations carried out are based on scientific criteria to a greater extent than in the past. Effective evaluation research must be capable of demonstrating objectively that social safety has changed as a result of a given measure, rather than simply indicating whether citizens or professionals are satisfied with the measure or believe that it has had an effect. While the use of a randomised experimental model is undoubtedly a worthy aim, it would be a misunderstanding to think that only such a model can deliver an insight into the effects of government measures. Situations in social reality can generally be manipulated to only a limited extent and a quasi-experimental study design (with its many variants) can itself already take us much further than at present. We have seen – especially in recent research – that with a little creativity it is 88

Evaluation_Safety.book Page 89 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in the Netherlands

certainly possible to devise a quasi-experimental design, including with retroactive force. The fact that a quasi-experimental design is frequently not chosen is due among other things to the costly and lengthy nature of such studies. Moreover, it is often decided to carry out an effect evaluation at a relatively late stage, and in particular in specific projects (special target groups or specific locations) it is then no longer always possible to employ an adequate research design. However, a superficial effect evaluation produces no usable conclusions about the effect of a measure on social safety and might therefore just as well not be carried out. The frequently heard view that if a number of poor-quality studies all point in the same direction this can provide a good impression of the effects is one which we do not support. Although the use of a quasi-experimental design is a necessary condition for a good-quality effect evaluation, it is by no means enough on its own. In addition to the internal validity, other factors are also important. For example, in the effect evaluations described in this report the validity of the definitions used was found to play an important role, especially where criminality was the outcome measure used (particularly with regard to measures designed to restrict opportunity). In particular in evaluations where an increase in the number of recorded crimes was observed after the introduction of a measure, it was often unclear to what extent this increase reflected an actual increase in crime. Several studies stressed that the increase was attributable to things such as increased willingness of victims to report offences, or to earlier detection and recording of incidents. It is therefore important to make use of diverse data sources as far as possible. There is also considerable room for improvement in the reporting of the evaluation studies carried out. In the evaluations which we examined, information was regularly missing about aspects such as how the measure had been introduced, the precise design of the study, the size of the sample, the moment of data gathering and the magnitude of the effects. It was also often unclear to what extent the results of the effect evaluations could be generalised for other situations: the external validity. Effect evaluations which were unclear on this point also usually had limited internal validity. In short, more knowledge about the effects of policy in increasing social safety can only be obtained by applying scientific criteria to impact evaluation research.

89

Evaluation_Safety.book Page 90 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

References FARRINGTON D.P., PETROSINO A., 2001, The Campbell Collaboration Crime and Justice Group, Annals of the American Academy of Political and Social Science. 587, 35-49. VAN NOIJE L., WITTEBROOD K., 2008, Sociale veiligheid ontsleuteld: veronderstelde en werkelijke effecten van veiligheidsbeleid, Den Haag, Sociaal en Cultureel Planbureau. SHERMAN L.W., FARRINGTON D. P., WELSH B. C., MACKENZIE D. L., (Eds.), 2002, Evidence-based crime prevention, London/New York, Routledge. SHERMAN L.W., FARRINGTON D. P., WELSH B. C., MACKENZIE D. L., (Eds.), 2006, Evidence-based crime prevention (revised edition), London, Routledge. SHERMAN L.W., GOTTFREDSON D., MACKENZIE D., ECK J., REUTER P., BUSHWAY S., 1997, Preventing crime: what works, what doesn’t, what’s promising, Washington, National Institute of Justice. TK, 2005/2006, Staat van de beleidsinformatie 2006, Tweede Kamer, 30558, nr. 2.

90

Evaluation_Safety.book Page 91 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales Tim Hope

This paper is concerned not so much with the substantive results of evaluation research concerning safety and crime prevention policies as with the question of the evaluability of community crime prevention, using examples from studies carried out in England and Wales. The concept of evaluability refers to those aspects of the institutional context of an intervention that affect the methodology of evaluation applied, and the validity of inferences drawn from applying that methodology. Unlike the discussion of methodology per se, the analysis of evaluability is a reflexive approach to criticism that seeks to give an account of how institutional context (the circumstances in which a particular evaluation is carried out) is related to the particular methodological approach adopted. The reflexive study of evaluability is far less developed than the study of evaluation method itself, yet the governance issues involved in evaluation research often have a crucial influence on its design, conduct and findings. Just as contemporary British governments have invested heavily in the evaluation of community crime prevention, so governance issues have had a critical impact on their evaluability.

Why evaluate? – The scientific case From the social scientific point of view, the classic statement about the purpose of evaluation research is Campbell (1969). Drawing upon the ethos of the ‘Great Society’ programmes of 1960s America, Campbell argued that an ‘experimenting society’ would be a ‘good society’ because social and political learning would be advanced by the application of scientific method. The use of scientific criteria to inform political choice would consequently lead to progress in the social policy sphere just as it seems to do in the field of medical science. Nevertheless, Campbell also saw the need to establish an experimental ethos in public administration for attaining this goal. The chief obstacle to establishing this ethos was not the absence of knowledge to experiment with, nor even the practical difficulties of implementing scientific methods of evaluation (which are the subject of evaluation research methodology), but the political obstacles in the way of informed social learning and choice stemming from the nature of governance itself. As 91

Evaluation_Safety.book Page 92 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

well as placing science as a methodological tool for governments to use, in Campbell’s vision of the reform society, science would also need to be a methodological tool with which to hold government accountable for its actions to a wider public interest. Without an external (publicly-accessible) method of accounting for political choice, public administrators have builtin incentives not only to avoid critical scrutiny of their public record and achievements but also for making political capital by claiming success for what they have purportedly done. Thus, the rigorous application of objective scientific method aspires to raise evaluation research above politics; providing evidence of What Works on a supposedly less partial and partisan basis than that of political discourse and competition.

Why evaluate? – The political case Usually in the United Kingdom, the type of community crime prevention intervention, where and on what it should be targeted, and who should benefit, are decided in advance by the sponsors. There are very few examples of ‘spontaneous’ community self-help and self-organisation. Aside from the academic study of ‘social movements’ (covering spontaneous community action), evaluation research is itself an applied research activity intimately tied-up with the process of government. Community crime prevention programmes are interventions by government into civil society and it is the nature of these interventions, and their sponsors’ needs, that affect their evaluability. In the United Kingdom, the sponsors of community crime prevention are usually the national government, and the purpose of these interventions is to implement its policies1. Thus, in answer to the question ‘why evaluate ?’ there are three 1.

This pattern tends to reflect the form of public administration in The United Kingdom. The United Kingdom is a unitary state, whose central government is accountable to the UK Parliament, consisting of three separate jurisdictions – England and Wales (containing the great majority of the population of the United Kingdom), Scotland, and Northern Ireland. Scotland, Wales and Northern Ireland possess their own devolved governments, though their statutory powers vary considerably both amongst themselves and vis-à-vis the United Kingdom government. Sub-nationally, there are separate, directly elected local governments and non-elected public authorities (including separate police authorities) that have limited tax-raising powers but cannot act ultra vires of national legislation. In all cases, the great majority of revenue required to fund local public services is collected by central government, and is subsequently disbursed to local administrations, with varying degrees of local government responsibility and central government fiscal control. In recent years, central government expenditure has been tied to public service performance regimes set by legislation (Hope, 2002).

92

Evaluation_Safety.book Page 93 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

motives or purposes that government sponsors have in commissioning or engaging with evaluation research: 앫 Accountability – the view that a government’s actions should be subject to some means of scrutiny accessible to those who elect it. Social scientific research methods are thus regarded as a superior means for ensuring political accountability. 앫 Technology – the view that the evaluation of governmental programmes will reveal information that will provide a ‘blue-print’ for what to do and how to do it. Social scientific research methods are thus regarded as a superior means for developing a technology of governance that will help with the engineering of social reform – that is, to discover ‘what works’. 앫 Validation – the view (often a political perception) that the purpose of evaluation is to provide proof of the desirability of a policy (and its sponsors) and/or its superiority vis-à-vis that of policy (and political) alternatives – that is, to demonstrate ‘what works’. Since evaluation is a political process, these three ‘political’ considerations have each played a crucial role in the conduct of evaluation research on community crime prevention. Distinctions in British policy and practice between them have become blurred and confused (Hope, 2002a), sometimes intentionally (Hope, 2004).

I.

The evaluation of safety and crime prevention policy in England and Wales

This paper is concerned with research in England and Wales aimed at evaluating community crime prevention, which is defined as interventions aimed at changing the social conditions that are believed to sustain crime and insecurity in people’s everyday environment, including public spaces and residential communities (Hope, 1995). Specifically, interventions often aim to change the institutional structure of local communities in ways that increase the exercise of social control both by private citizens and the authorities (Hope, Karstedt, 2003; Hope, 1998; 1995; Foster, Hope, 1993). ‘Institutions’ and ‘social control’ can be defined as both informal and formal – examples of which are given in Table 1. In essence, these are interventions that address residential places (neighbourhoods) as holistic entities (communities) which encapsulate their own internal social relations and dynamics, including the interaction of formal and informal institutions to ‘co-produce’ safety in common environments (Hope, 1995). As such, they are Comprehensive Community Initiatives – CCIs (Connell, Kubisch, 1998).

93

Evaluation_Safety.book Page 94 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

Institutions

Social control

Formal 앫 police 앫 government 앫 law enforcement 앫 environmental design 앫 situational crime prévention

Informal 앫 families, networks 앫 community associations 앫 natural surveillance 앫 invocation of authority 앫 informal social control 앫 socialisation

Table 1: Dimensions of Community Crime Prevention

1.

Types of programmes

There have been two generic kinds of central government programme with potential to affect community safety: i) Crime Prevention. Large-scale programmes in England and Wales have included the Safer Cities Programmes and the Crime Reduction Programme, along with funds available through regional programmes to fund local authority projects, since 1998 via the local government machinery of Crime and Disorder Reduction Partnerships (Hope, 2005a, 2002b). These programmes have included the range of activities described in Table 1. Notable investments have been in Neighbourhood Watch, public ClosedCircuit Television (CCTV), and Street-Lighting. There has also been a steady investment in community policing, including Problem-Oriented Policing, Reassurance Policing and, most recently, Neighbourhood Policing. In most of these funding programmes, the projects have been small-scale ‘demonstration projects’ targeted on local neighbourhoods. In general, authorities seek to deliver specific crime prevention services to communities. Sometimes these are evaluated for cost-effectiveness, and sometimes they have been accompanied, or comprise part of, national programme-level evaluations. A common governance theme is partnership – between central and local government, between governmental agencies (e.g. local government and police services), and between these bodies and civil society (whether private citizens or corporate bodies). Nevertheless, there are also power structures, hierarchies and organisational cultures amongst agents that affect how they interact together ‘in partnership’ and how and what outcomes are produced. ii) Urban Regeneration. Community safety has also been one of the objectives of urban regeneration programmes, though it has not been an exclusive or sole objective. In contrast to the direct effects of programmes

94

Evaluation_Safety.book Page 95 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

delivering specific crime prevention, regeneration programmes tend to address community conditions that give rise to crime – social exclusion, powerlessness, lack of collective efficacy and isolation from mainstream society – thereby setting in train community-level change that leads to greater social control (Hope, 2005a). From the 1970s onwards, regeneration programmes have become increasingly focussed and specific, moving away from main-stream funding towards more targeted demonstration programmes. National programmes have included: the Priority Estates Project (PEP), the Design Improvement Controlled Experiment (DICE), the Single Regeneration Budget programme (SRB), and most recently, the New Deal for Communities (NDC) and Housing Market Renewal (HMR) programmes. During the course of this development, community safety has become an increasingly salient, and distinctive, part of the package of issues to be addressed.

2.

Evaluation research into community crime prevention

The history of evaluation research in community crime prevention mirrors the development of crime prevention policy and practice in the UK. Key evaluation studies have included2: 앫 The Home Office Crime Prevention Feasibility Study. This was aimed at demonstrating the application of systematic (evidence-based) prevention and multi-agency partnership working (Hope, Murphy, 1983; Gladstone, 1980). The research utilised an action-research, casestudy methodology. 앫 Neighbourhood Watch. An evaluation of the impact of a pilot Neighbourhood Watch programme in two residential neighbourhoods, utilising a quasi-experimental research design (Bennett, 1990). This study also introduced the use of social survey data as a source of measures of impact (see also studies 3, 4, and 6). 앫 Community Policing (Fear Reduction). A quasi-experimental evaluation of a policing initiative aiming to reduce fear of crime and improve the quality of life of residents on two residential estates (Bennett, 1991). 앫 Priority Estates Project Evaluation Study. A quasi-experimental evaluation of the effect on community safety of a housing environmental and management (urban regeneration) programme on two estates in two different English cities (Foster, Hope, 1993).

2.

This is a selective rather than comprehensive list, selected to represent key issues or developments in evaluation methodology, discussed in Part 2.

95

Evaluation_Safety.book Page 96 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

앫 The Kirkholt Burglary Reduction Project (Repeat Victimisation). A multiagency initiative, including situational, offender-based, and social prevention measures (Forrester et al., 1990; 1988). It has since become famous for pioneering the prevention of repeat victimisation and for being considered an early exemplar of ‘realist’ evaluation (Pawson, Tilley, 1997). The Kirkholt Project has spawned a number of evaluations in the UK and elsewhere focusing on repeat victimisation prevention (Farrell, 2005). 앫 The Safer Cities Programme. A multi-agency, multiplex, multi-site crime prevention programme. Phase 1 ran from 1988-1995, funded and managed by the Home Office; (Phase 2 ran between 1994 and 1998 funded and managed by the Ministry for Local Government). The Programme itself was monitored administratively (Knox et al., 2000). A key focus of Phase 1 was the prevention of burglary: using Geographic Information Systems (GIS) technology and micro-econometric modelling techniques, Ekblom et al. (1996), carried out probably the most statistically sophisticated study in an effort to measure programme intensity and incorporate selection bias (see below). Other, smaller scale evaluation studies also focused on burglary, though using mainly qualitative techniques within a ‘realist’ approach (Tilley 1993; Tilley, Webb, 1994). 앫 Design Improvement Controlled Experiment (DICE). Carried out between 1989 and 1997, DICE consisted of radical design improvement schemes on selected council housing estates in England, the evaluation used mixed methods (including surveys) to evaluate the impact, costs and benefits of the schemes3. 앫 Communities that Care (CtC), 1998-2003. In the mid-1990s the Joseph Rowntree Foundation (JRF) funded the Communities that Care (CtC) prevention initiative. This early intervention programme targets children living in communities and families that are deemed to put them at risk of developing social problems. The CtC approach focuses on small geographical areas and involves bringing together local community representatives, professionals working in the area and senior managers responsible for service management. Participants are given training, and provided with evidence of the levels of risk and protection in their community. From this they design an action plan that seeks to enhance existing services or introduce new ones likely to reduce risk. CtC is therefore not simply a service delivery programme, but a process leading to the identification of a programme of work, and a method of facilitating the delivery of well-co-ordinated services that 3.

http://www.communities.gov.uk/archived/general-content/citiesandregions/221449/

96

Evaluation_Safety.book Page 97 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

reduce risk and increase protection. CtC does not deliver services itself, but facilitates and activates change in a local area. In March 1998, JRF funded three demonstration projects in the UK to test whether this approach could be successful… A research team from the University of Sheffield was commissioned to undertake an evaluation between 1998 and 2003 (Joseph Rowntree Foundation, 2004: 1; Crow et al., 2004). 앫 The New Deal for Communities Programme (NDC), 1998-2007. The NDC Programme is a key element of the Government’s National Strategy for Neighbourhood Renewal. The Programme targets 39 neighbourhoods across England with some of the highest levels of multiple deprivation. Key objectives of the Programme address the five overarching goals of the National Strategy, which are to: reduce worklessness; reduce crime; improve health; improve skills; and improve housing and the physical environment. A National Programme of Evaluation of the NDC has been carried out, using a range of indicators and (non-experimental) analytic methods. Evaluation studies have covered the range of NDC goals including crime reduction (e.g. McLennan, Whitworth, 2008). 앫 The Crime Reduction Programme (CRP), 1999-2002. This was the largest programme of community-based crime prevention ever attempted in the UK, with an original budget of £250 million (315 million euros, approx. at the time), allocated mainly to local projects, focusing upon different, thematic, crime problems, with 10 per cent ear-marked for independent evaluation studies of cost-effectiveness (Maguire, 2004). Apparently, more than 80 evaluation reports were produced, though fewer than that have been published (Hope, 2008a). The most intensively evaluated component of the CRP was the Reducing Burglary Initiative – Phase 1 (Hope et al., 2004; Hirschfield 2004; Millie, Hough, 2004; Bowles, Pradiptyo, 2004; Kodz et al., 2004). The CRP also contained a programme of around £150-£200 million (189 – 252 million euros, approx.) for CCTV systems in local areas, which has also been evaluated (Gill, Spriggs, 2005). 앫 Community Policing (Reassurance Policing/Neighbourhood Policing). Reassurance Policing is a model of neighbourhood policing which seeks to improve public confidence in policing by involving local communities in identifying priority crime and disorder issues in their neighbourhood which they then tackle together with the police and other public services and partners. (http://www.crimereduction.homeoffice.gov.uk/ policing17.htm). There was evaluation of the outcomes of the National Reassurance Policing Programme (NRPP) in England between 2003/04 and 2004/05 (Tuffin et al., 2006). …The implementation of neighbourhood policing has represented a significant undertaking for the Government, police service and their partners. Neighbourhood policing 97

Evaluation_Safety.book Page 98 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

was initially piloted at a ward level as part of the [NRPP]. Following [this] the three-year Neighbourhood Policing Programme (NPP) was officially launched in April 2005, and has sought to deliver on the Government’s commitment for every neighbourhood in England and Wales to have a neighbourhood policing team by 2008 (Quinton, Morris, 2008, iv)4.

II.

The evaluability of community crime prevention

1.

Evidence-Based Policy and Practice?

British central governments have invested heavily in crime prevention over the past Quarter-Century. Since 1997, New Labour Governments have specifically coupled crime prevention (or, as they call it, Crime and Disorder Reduction) with the ‘ideological practice’ of Evidence-based Policy and Practice (EBPP), encapsulating the policy pursuit of ‘What Works’ (Goldblatt, Lewis, 1998). The public administration of EBPP is to be a matter, firstly, of accumulating expert knowledge and then having that expertise adopted and applied more generally. The ideal model is of an influential and credible government, guided by expert knowledge, ‘demonstrating’ by practical example that certain programs ‘work’ and thereby persuading others to adopt them (Hope, Karstedt, 2003). Tilley and Laycock (2000), for example, describe the research and implementation programmes involved in the British Government’s development and dissemination of ‘evidence-based’ crime prevention policy. Increasingly, the results of evaluations are to be aggregated and assessed by systematic review, whose role is to act as a filter and qualitycontrol for implementing interventions, based upon methodological criteria concerning the ‘quality’ of the evidence available from completed evaluation studies. One such set of criteria – the Scientific Methods Scale (SMS) (Farrington, 2003) – has become increasingly influential as an ostensible criterion for central government evaluation research (Hollin, 2008; Hope, 2005c). The adoption of community crime prevention practices locally is also a continuation of project-based demonstration. Thus, once persuaded, those with the authority to act are enjoined further to establish and evaluate their own ‘projects’ so that they too can demonstrate benefits, both to their constituents as the recipients of prevention, and to the public purse, that cost-effectiveness has been achieved. The mode in which prevention 4.

An early-stages evaluation report is available.

98

Evaluation_Safety.book Page 99 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

projects are enjoined to operate conforms, in most respects, to a model of rational strategic planning (Hope, Karstedt, 2003; Hope, Sparks, 2000). Knowledge of a particular crime problem is to be ascertained in advance of trying to implement action, and the selection of services and measures to be delivered is then tailored to the key features of crime manifest in the specific target area or group. The model of EBPP combines the governance precepts of the ‘New Public Management (NPM)’ (Hope, 2002b) with prudential governmental action against social risk (O’Malley, 2000). The method of demonstration by practical example is believed to work because those who have the responsibility and authority to take action will be shown how to identify prospective benefits accruing in their self-interest, as articulators of the interests of their constituents or ‘customers’. They will be thus persuaded to take requisite action to maximise the utility of their policies on the basis of a calculation of the cost-effectiveness of individual preventive interventions (e.g. Sherman et al., 2002). Syntheses of the evidence from evaluation research are used to develop practical advice, guidance and ‘tool-kits’ for local practitioners5. The practical problems of rigorous social experimentation are well known: such ‘quasi-experiments’ may be difficult to set-up, individual experiments may become un-representative, or may involve unethical choices affecting ‘subjects’, the random administration of penalties, and the denial of beneficial treatments. The problem for policy-makers, though, is a different one: the Martinson Problem (Pawson and Tilley, 1997) or, in other words, that programmes may not always be judged to be successful despite the ambitions of their sponsors. For the policy-maker, the ‘threat’ posed by evaluation research is also different than for the social scientist. Social science is oriented towards the avoidance of Type I Error: that is, to avoid concluding that a finding is true when it is not. For the policy-maker, however, the situation is reversed: although policy-makers who make Type I errors may well incur opportunity costs to the wider public interest (i.e. by diverting resources towards ineffectual programmes and away from desirable policies), the direct political costs of making alternative Type II Errors (i.e. to conclude that a finding is false when it is true) are much greater: risking throwing away their programmes and their credibility needlessly. Thus, evaluation research always contains the potential to threaten the political capital vested in social programmes. Evaluation research embodies a political dilemma: that the price to be paid for insisting on methodological rigour on the grounds both of scientific quality and democratic accountability may well be a lack of utility for policy-makers (Hope, 2002a). The Martinson Problem (arising from an 5.

e.g. http://www.crimereduction.homeoffice.gov.uk/

99

Evaluation_Safety.book Page 100 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

earlier era of evaluation of sentencing and penal treatment) is that the rigour of evaluation research is perceived by policy-makers to heighten the probability of reaching ‘Nothing Works’ conclusions. What may serve the public interest in one way – i.e. by holding government accountable – may hinder it in another – i.e. by inhibiting government from finding out “what works”, thereby inhibiting political progress. There would thus be considerable advantage to be gained if it were possible to preserve the safeguards of validity inherent in ‘scientific method’ while at the same time producing useful results of ‘what works’. Nevertheless, when push comes to shove, the greater resources of persuasion available to politics than to science can mean an uneven contest in which methodology comes to be sacrificed to political expediency (Hope, 2004). For all its promises and inducements to social scientists, the ideology of EBPP can be used as a means of exerting political control over the conduct of evaluation research in order to validate and legitimise government policy (Hope, 2008b). EBPP invokes the aura of science to gain support and consent for policy (Campbell, 1969). In this sense, the ‘evidence’ generated from evaluation research, resting as it does on apparently scientific foundations, can be used politically as an unassailable means of supporting Government policies (Hope, 2006). As experienced during an evaluation of part of the Reducing Burglary Initiative of the Home Office Crime Reduction Programme (1999-2003), this stratagem puts pressure on evaluation researchers to produce the ‘correct’ findings (STC, 2006), leading contracted researchers to compromise on research quality standards; and government to commission alternative congenial research (Hope, 2004) and/or to ignore, distort, or suppress the findings of the research that it has commissioned (Hope, 2008a).

2.

The logic of evaluation research

The purpose of evaluation research is causal attribution – specifically, to assess the extent to which the deliberate implementation of policy intervention X changes substantially an identified crime/security phenomenon Y. Attaching a value to this assessment (cost-effectiveness) is an important, though secondary, purpose since it depends upon the identification of the scale of effect. The methodological task is to identify the unique effect of the intervention, net of other influences. Thus, evaluation methodology has the two-fold task of estimating the unique effect of an intervention, while ensuring that such estimates are valid representations of true effects, primarily by protecting against bias in the production of the estimates.

100

Evaluation_Safety.book Page 101 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

The simplest conception of an evaluation is in the form of a ‘black box’ experiment (see Figure 1), where a particular intervention is conceptualised as a ‘treatment’ applied to a group of subjects and an estimate is obtained of the mean effect of the treatment on the subject group. The treatment is causally prior to the effect.

Figure 1: Black-Box Experiment

Evaluation methodology has evolved to solve the fundamental evaluation problem that this conception poses. As Heckman and Smith (1995, 87) put it, this arises from: …the impossibility of observing what would happen to a given person [or community] in both the state where he or she receives a treatment (or participates in a program) and the state where he or she does not. If a person could be observed in both states, the impact of the treatment could be calculated by comparing his or her outcomes in the two states, and the evaluation problem would be solved Evaluation methodology should thus be concerned with the operationalisation of methods that support the valid estimation of the effect of being subject to an intervention, net of alternative, counter-factual conditions that the subject could experience. Empirically, we only have available the observable ‘fact’ of being subject to the intervention. Since we cannot observe what would have happened if the intervention had not happened (i.e. the counter-factual condition), we have no certain means of knowing what factors ‘caused’ the specific conditions observed in the subject during and after the intervention. Because the counter-factual is unobservable, we cannot evaluate the actual probability of the effect occurring against the un-actualised probability of it not occurring. In the experimental design (Figure 2), those persons assigned to nonexposed conditions represent the counterfactual condition because the process of random assignment to the ‘exposed’ and ‘non-exposed’ conditions means that members of either group are ‘exchangeable’ and that the subsequently observed differences between the two represents the 101

Evaluation_Safety.book Page 102 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

difference between the factual and the counterfactual, and hence indicates causation (Hernán, Robins, 2006). It should be noted that ‘random allocation’ to the control condition does not imply that the experience of the control condition is itself ‘random’. Unlike in a laboratory, evaluation research applied to social policy interventions takes place in ‘field-settings’ where both the experiences of the subjects and the intervention are fundamentally embedded in social life, hence the term quasiexperimentation (Cook, Campbell,1979). Random allocation, therefore, cannot create a vacuum by disembedding subjects from their social context; rather, in principle, it ensures that subjects are exchangeable (i.e. have identical experiences) so that observed differences can only be due to the unique experience of the intervention, which is thus an abstracted aspect of experience that only makes sense in terms of the logic of the methodology used to abstract it.

Figure 2: The Experimental Model

Yet if randomisation is crucial to the capacity to control the estimation of effects, then it is also critical to be able to control randomisation. Community crime prevention interventions differ from criminal justice interventions in terms of the styles of governance associated with the different institutional settings and programmes typically implemented within each sphere (Hope, 2005c). In practice, this shapes the ability to exercise control in order to bring about subjects’ compliance with the intervention, and hence defines the nature of the intervention itself. Community interventions tend to be premised upon mechanisms of collective, voluntary participation – for example, where a community

102

Evaluation_Safety.book Page 103 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

group, or non-state agency acting on its behalf, is concerned with collective self-advancement; or where a government is concerned with the physical, economic or social ‘regeneration’ of a residential area. In contrast, the ‘treatment’ programmes taking place in courts and corrections (including supervision in community settings) are characterised by individual, constrained participation. Typically, they are concerned with altering the rewards and/or sanctions bearing upon individuals so that they can ‘fit-in’ to prevailing institutional arrangements. Here, ideas about intervention arise from different spheres – of criminal justice, education and ‘correction’ – that are, at root, the responsibility of the State, the institution with legitimate, judicial powers of coercion over subjects. Subjects’ participation in community programmes is voluntary because these involve the social and institutional arrangements of civil society. The mode of governance in civil society is indirect and diffuse: holders of power (government) have to govern ‘at a distance’, influencing and shaping the setting itself, utilising a range of essentially non-coercive measures, in order to engineer the compliance of private citizens, including influencing the future behaviour of ‘putative’ as distinct from actual offenders or victims6. In contrast, settings such as courts, prisons and (only to a lesser extent) schools, and even labour markets (particularly of the unemployed or low waged), include subjects whose participation is largely involuntary or over whom power-holders (essentially the state and its agencies) have some considerable (ultimately coercive) power of control and influence7. These latter settings differ therefore from civil society communities in that they comprise the various arenas where there is a greater level of resource for engineering the compliance of subjects available to agencies with the authority and legitimacy to bring about change. The capacity to control subjects directly is crucial not just because it shapes the policy conditions that define the intervention but also because it affects the research conditions that will define the most appropriate evaluation research methodology. Two conditions of control are crucial: the ability to (a) allocate subjects to specific research conditions, and (b) command the compliance of agents in implementing policy interventions (treatments). The criteria employed to assess methodological quality in the Scientific Methods Scale (SMS, noted above) rely principally upon the power of 6.

7.

That is, they accord to the sphere of ‘primary crime prevention’ “identifying conditions of the physical and social environment that provide opportunities for or precipitate criminal acts (Brantingham, Faust, 1976, 290). Similarly, these accord to the sphere of ‘secondary crime prevention’ (Brantingham, Faust, 1976).

103

Evaluation_Safety.book Page 104 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

control. For instance, a systematic review based upon the SMS quality standards, under the heading “what doesn’t work?” listed …community mobilization of residents’ efforts against crime in high-crime, inner city areas of concentrated poverty (Sherman et al., 1998, 8), citing Hope (1995) as corroboration (erroneously) of this proposition. Yet because of the differential availability of coercive control in community relative to criminal justice settings, the application of the SMS review methodology does not tell us to what extent this conclusion is due to the problem of operationalising SMS-favoured methods (of randomisation and control) rather than the relative efficacy of each sphere to intervene in crime. Worse, comparatively unfavourable inferences could be drawn simply on the basis of comparing flawed efforts at experimentation in community settings with competently designed and well-executed experiments in settings more amenable to the manipulation and control of human subjects (Hope, 2005c).

3.

Methodology of evaluation research

Quasi-experimental methods have evolved to compensate for the difficulties of randomisation-by-design encountered by applying experimental designs in field-settings (Figure 3). Essentially, they rely upon statistical adjustment to compensate for the absence of the capacity to control by design (Judd, Kenny, 1981). The evaluations of Neighbourhood Watch (Study 2), Community Policing (Study 3), Priority Estates Project (Study 4), DICE (Study 7) and Community Policing (Study 9) have all employed a similar design: non-randomised matching of ‘experimental’ and ‘control’ sites, using regression adjustment methods applied social survey data to estimate treatment effects, net of the effect of measured covariates (i.e. social characteristics of residents correlated with the outcome variable) that might differ between experimental and control sites (Farrington, 1997).

104

Evaluation_Safety.book Page 105 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

Figure 3: The Quasi-Experimental Model

The methodological contribution of Campbell and his followers lay in advocating research design solutions, short of the pure experiment, to the many various ‘threats’ to the validity of programme evaluation research (Cook, Campbell, 1979). Essentially, these threats stem from the practical difficulties of implementing valid experiments with human subjects in field-settings. In this tradition there are four classes of threat to validity (Farrington, 2003): 앫 Statistical conclusion validity: whether the hypothesis of a causal relationship between intervention and outcome is supported by inferential statistical tests. As in all statistical hypothesis testing, there are two test criteria: testing the probability that the intervention hypothesis should be accepted (Type I Error testing); and testing the probability that the null hypothesis should be rejected (Type II Error). Information to assist in these decisions rests, respectively, upon the calculation of statistical significance (via the appropriate statistical test) and of the size of effect (statistical power calculation) (Farrington, 2003). 앫 Internal validity: the key question of inference of a causal relationship between intervention and outcome. 앫 Construct validity: …the adequacy of the operational definition and measurement of the theoretical constructs that underlie the intervention and the outcome (Farrington, 2003, 54). 105

Evaluation_Safety.book Page 106 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

앫 External validity: the degree and extent to which we can have confidence that the results from a particular intervention can be generalised to other contexts. While there has been much debate about the way in which evaluation methods can address these issues, these classes of validity still remain a clear categorisation of the nature of the methodological problems encountered in evaluation research.

4.

The realist critique

A crucial difference between a community crime prevention initiative and, say, a controlled laboratory experiment is that, in the former, the ‘treatment’ has far less integrity a priori. In field-settings, any ‘treatment’ intervention is produced over time in specific contexts, in complex practical and organisational ways, utilising varying combinations of authority, capital and resources. Moreover, any intervention always acts upon a specific context (such as a residential community) and produces its effect often through a variety of mechanisms intrinsic to the intervention. Study 3 (Bennett, 1991) was criticised by Pawson and Tilley (1994) on the basis that its ‘black-box’ model of experimental intervention may have concealed a variety of mechanisms within the intervention that might have an effect on the outcome; while the statistical discounting of covariate differences between sites conceals the likelihood of different contexts having separate effects on outcomes (see also Bennett, 1996; Pawson, Tilley, 1996). Essentially, their critique is concerned with establishing the greater importance of construct and external validity, which they suggest has greater political and practical relevance, than social scientists’ primary concerns with internal and conclusion validity. Nevertheless, there are incommensurate and contradictory trade-offs amongst the four types of validity (Judd, Kenny, 1981). Thus, the critique of the Random Controlled Trial (RCT) applied to field-settings is that although it maximises Internal and Conclusion Validity, it risks both greater artificiality and oppressive/unethical practice (usually both together). Further, in being a practical intervention, in order to implement an RCT, it is necessary to abstract it artificially from the social and institutional contexts in which its subjects are embedded. Thus, the dilemma of the RCT is that the more it maximises internal validity the more it looses on external and conclusion validity; the dilemma for the realist critique is the opposite.

106

Evaluation_Safety.book Page 107 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

Pawson and Tilley (1997) propose an alternative way of conceptualising the production of an outcome by hypothesising that any outcome is always the product of a particular context-mechanism-outcome (CMO) configuration (Figure 4). Realist evaluation is based on a comparative case-study methodology (e.g. Tilley, Webb, 1994). The first step is to construct a set of plausible Context-Mechanism-Outcome (CMO) configurations that would typify the crime prevention project in question. Second, further cases (crime prevention projects) are sought that replicate the initial case, except with some known variation in context, mechanism or outcome. Through the accumulation of case studies (a process of replication), systematic variations in CMO configurations can be discovered, that can then be generalised as policy-prescriptions. Tilley (1993), for example, reports two case-studies that purport to ‘replicate’ the Kirkholt Burglary Reduction Project, as part of a process to assess the effectiveness of a policy of repeat burglary prevention (Pawson, Tilley, 1997). Further replications are also held to vindicate the Kirkholt thesis (Farrell, 2005).

Figure 4: Realist Experiment

Nevertheless, this so-called scientific realist method threatens the internal and conclusion validity of its findings. In particular, it encourages (or insufficiently guards against) bias in its selection of case studies and in its

107

Evaluation_Safety.book Page 108 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

investigation of causal mechanisms. These difficulties stem both from its methodology, that is, a reliance on qualitative, case-study methods – and its epistemology – that is, its inductive method of enquiry. For example, ‘replication’ evidence from samples of CMOs (Hope, 2002a), analysis of national trends (Hope, 2007) and evaluation studies (Hope et al., 2004) – have found no further instances to date of the mechanism thought to be central to the apparent success of the Kirkholt Project – that a change in either the prevalence of repeat victims, or the frequency rate of (repeat) victimisation, has any significant association with a change in the overall crime (incidence) rate. If anything, it would seem that the crime prevalence rate (the proportion of victims in the population) has a far greater impact on the crime incidence rate than does the frequency (repetition) rate of victimisation (Hope, 2007; 2002a). The ‘realist’ interpretation of the success of the Kirkholt Project is ex post facto. This opens up ‘realist’ evaluation to the charge of selective interpretation of evidence (a fallacy of the inductive method of reasoning), which is not helped by the absence of research design and comparison in the case-study method commonly employed. One issue concerns that of intent – without establishing what was intended prior to implementation, it would not be possible, logically, to establish that what had happened was what had been intended. There is always the danger of the ex post facto fallacy (post hoc, ergo propter hoc), which is particularly worrying in policy evaluation: if original intentions are not met, it may be politically attractive to take credit retrospectively for what had actually happened fortuitously, irrespective of any particular intention (Campbell, 1969). These problems are compounded by two well-known problems of selectivity (Hope, 2006; Hope et al., 2004): 앫 Selection bias: the difficulty of ‘control’ in community crime prevention (noted above) means that projects to be evaluated are never selected at random; more usually, they have been selected to participate on some basis that may be related to the outcome. Candidates for evaluation may then have already been ‘cherry-picked’ for inclusion on their likelihood or promise of success (e.g. if crime has been going down previously, or if the implementing agents seem enthusiastic, co-operative, competent and efficacious). The problem is compounded, first, when projects are selected on a competitive basis; and second, when they are given a ‘prelaunch improvement’; both of which occurred during the Reducing Burglary Initiative (Study 8) (Hope, 2004)8. A further consequence of 8.

And since, as it turned out, neither effort actually guaranteed success, sponsors were simply prompted to re-analyse the data so as to come up with sought-after outcomes (Hope, 2004).

108

Evaluation_Safety.book Page 109 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

EBPP is that it justifies the reporting only of the selection of ‘successful’ case studies, biasing the result obtained and inhibiting learning (which may need to analyse failure as much as success). 앫 Regression to the Mean: national crime prevention programmes may be targeted on high crime areas, on the basis that a greater preventive gain might be attained relative to expenditure, as was the case with many projects in the Crime Reduction Programme (Study 10). That said, it is then difficult to distinguish the extent to which any subsequent reduction is due merely to the phenomenon of regression to the mean, at least by the methods usually employed (Hope, 2006; Campbell, 1969). A related logical problem is that of expectation. The customary antonym in crime prevention policy to “what works” is “nothing works” – that is, the expectation that action will either have a positive benefit (crime reduction) or a nil-effect. What this antonym cannot incorporate is the possibility of unintended but still nevertheless significant negative or harmful consequences as a result of crime preventive intervention. To the extent that those who propose crime-reductive actions are oriented towards political utility, the logical possibility of unintended consequences – that actions might cause harm or are more costly than the problem they sought to remedy – cannot be countenanced without undermining their rationale of crime prevention. This mode of thought does not allow for the logical possibility of crime permission: that the specific actions of the programme, regardless of whether or not they are intended to reduce it, may nevertheless permit or encourage crime to increase. Nevertheless, evidence is available from the Reducing Burglary Initiative (Study 8) that the particular activities of a particular crime prevention project – ironically focussing specifically on repeat burglary prevention – appeared to have brought about a 38 per cent increase in the rate of burglary victimisation, as well as in the rate of repeat victimisation (Hope et al., 2004)9. In sum, the realist model fails to account for both political selectivity and social selectivity and thus becomes vulnerable (whether intentionally or not) to bias (Figure 5).

9.

Another occasion for retrospective re-analysis (see fn. 6).

109

Evaluation_Safety.book Page 110 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

Figure 5: The Realist Model

5.

Programme Mechanisms

Despite its vulnerability to bias in selection and interpretation, the realist approach, with its emphasis on the production of prevention, does point to the importance of mechanisms – or rather causal sequences – developing over time to generate crime prevention outcomes, which the typical ‘black-box’ experimental approach fails to capture. Theories of the generation of community crime prevention, based on ideas about collective efficacy and social capital, posit causal sequences whereby contextual variables interact with intra-community social processes to produce outcomes developing over time (Hope, Karstedt, 2003; Hope, 1995). Such processes were identified by the ‘mixed methods’ approach of the Priority Estates Project Evaluation Study (Study 4) (Foster, Hope, 1993). Social capital theories posit a set of necessary interaction effects in order to produce community safety; but it is precisely these kinds of interaction effect that the assumptions of the SMS quality criteria would

110

Evaluation_Safety.book Page 111 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

construe as ‘threats to validity’10. Broadly, these take the form of two kinds of so-called threat: 앫 Selection x Treatment. This could itself take two forms: first, where the intervention programme cannot be separated from the social and political structures in which target communities are embedded. For example, only certain kinds of community are likely to support ‘neighbourhood watch’ community organisations, thus state-agencies interested in establishing them will select only those kinds of communities that provide fertile soil for neighbourhood watch groups to flourish, while social processes within such communities will also be ‘working’ to capture state resources, creating ‘club goods’ of security for themselves (Hope, Trickett, 2004). Arguably, the nil effect identified in evaluation studies of Neighbourhood Watch (Study 2 above) may be due to the similar levels of indigenous social capital present in the matched experimental and control communities. Second, as a consequence, a kind of ‘elective affinity’ (to use Weber’s term) is established between treatment and selection, so that the one cannot operate without the other, to the extent that the treatment itself comes to be defined in terms of the selection criteria (i.e. the characteristics of the setting). 앫 Treatment x Outcome. The objective of the SMS quality scale is to ensure that it is the treatment that causes the outcome – termed Conclusion I (Judd, Kenny, 1981). However, there are two other relations between treatment and outcome that, although they more accurately define the process of change, through a sequence of mediating processes, paradoxically reduce the correlation between treatment and outcome that SMS designs endeavour to maximise, for instance, Conclusion II where the treatment causes the potential mediator. Here, the treatment intervention into a community may actually seek to enhance the ‘mediating variable’ – for example, the development of the right mix of social capital. Thus, the intervention creates opportunities for bridging capital to be developed, say, by creating intra-community organisations, utilising and empowering community leaders (already possessing ‘linking’ social capital), which then empowers the community by creating additional linking capital for community members, that enhances their efforts to create bridging institutions, and so on (Hope, Karstedt, 2003). There is also Conclusion III – that the sum of these 10. In the evaluation research literature these are often referred to as ‘moderator’ and ‘mediator’ effects on treatment (Farrington, 2003; 1998), though in individualtreatment interventions these appear not to be as severe a set of ‘threats’ as they do in community crime prevention.

111

Evaluation_Safety.book Page 112 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

activities (e.g. collective efficacy) itself becomes the mediating variable, such that when it is controlled statistically, or in research design, the direct correlation between the treatment variable (the originating governance intervention) and the outcome variable (i.e. crime) disappears, even though it has actually had an important, though indirect, causal influence. Evidence of both these ‘threats’ to explanation is apparent from Study 4 (Hope, 2005c). Thus, the paradox is that were the evaluation research design to control for these apparent threats to validity – or that an SMS review rejected the evidence from studies that did not seem to have these qualities – it would be likely to reject the influence of the treatment on the outcome when, in practice, it had a very significant effect. In this case, it would be the evaluation method itself, and not the intervention, that produced the result, thus leading to Type II error inferences – that the treatment had no effect, when in fact it did. A related issue in community crime prevention is that of spill-over – whether displacement of crime or diffusion of benefit. Space precludes a detailed discussion of measurement issues involved, though again, it is necessary to consider methodological issues concerning the validity of observations of apparent spill-over effects. For instance, Hope et al. (2004 – Study 8) found, paradoxically, that more projects achieved a greater net reduction in crime in their wider area, including a surrounding ‘buffer zone’ than in their specific target area. Yet this was more likely to be a spurious rather than a true diffusion of benefit since projects were actually part of bigger crime reduction initiatives covering wider areas beyond the target, and that they were likely to affect the behaviour of populations of offenders, victims and community members in the communities beyond their relatively small target areas.

6.

Programme implementation

The realist critique also points to the importance of intervention process in the estimation of causal effect. The process of implementation is not only crucial to a project but also a logical pre-requisite to evaluation – whatever the rigour of evaluation methodology, it will not be possible to detect the effects of an intervention if little or nothing has been implemented to produce those effects (Hope, 1985). Generally, outcomes are likely to be affected by the 앫 Efficacy of crime prevention measures (this is the traditional concern of programme developers);

112

Evaluation_Safety.book Page 113 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

앫 Tractability of the target/community context: i.e. how easy or difficult it is to implement the crime prevention measures, given the characteristics of different environmental and community contexts; and 앫 Efficiency of the organisation of delivery: i.e. how efficient or productive are the agents and their procedures in implementing the intervention. Each of these independent factors is likely, separately and in combination, to interact to influence the probability of a specific outcome of an intervention (Hope et al., 2004). Rather than being separate issues, the evaluation of implementation is necessary condition for the evaluation of outcome. Yet it is also necessary to separate out its effects in order to ascertain programme outcomes, or more usually failures, adducible to failures of implementation (Hope, 2005b). Implementation issues have arisen in most British community crime prevention programmes (Hope, Murphy, 1983). This has lead to two methodological responses: 앫 Process studies of implementation: evaluation studies of the CRP Reducing Burglary Initiative (Study 8) were obliged to undertake detailed process studies, moving-on from mere description and monitoring in order to explain specific implementation outcomes, provide inputs to cost-effectiveness studies (Bowles, Pradiptyo, 2004) and generate a set of conditions that might account for differential impacts. For instance, the relative failure of projects to reduce burglary was related to a differential willingness to abandon and adapt initial intervention plans, which, in turn, was related to the organisation of the project and the characteristics of the lead agency. In particular, projects that were dominated by the police seemed notably inflexible, persisting with ineffectual implementation plans that eventually lead to failure (Hope et al., 2004). 앫 Measurement of project intensity of action: following pioneering work by Ekblom et al. (1996, Study 6), evaluation studies in the RBI (Study 8) developed various quantitative measures of project intensity of action, which could measure the amount, quantity, duration and tempo of implementation. Hope et al. (2004, Study 8) combined both of the above approaches to create an Intensity of Action score – consisting of a periodic, quantitative, cumulative measure of the impact of implementation during and beyond the duration of the project. Observational data on implementation, administrative and cost data were combined into a Calendar of Action from which a quantitative measure could be derived, which was immensely useful in econometric, time-series analysis of programme impact (Crawley, Hope, 2003).

113

Evaluation_Safety.book Page 114 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

Conclusion: heterogeneity and selection bias The realist critique both illustrates and succumbs to two methodological problems of non-experimental research that have a major effect on the inferences to be drawn from evaluation research: heterogeneity and selection bias. Contexts, interventions and outcomes are heterogeneous (as the realists argue) but non-experimental evaluation research also needs to build-into its design and analysis protection against inferential error due to manifold selection bias (of which realists are wholly guilty). As both interventions and evaluations are embedded in the social world, quasiexperimental research design is insufficiently powerful on its own to isolate generalisable estimates of programme effect free from selection bias. Building on the work of Heckman (2001) in micro-econometrics, evaluation research needs to model selection effects, isolate unique programme effects, and identify heterogeneous outcomes (Figure 6). Hope et al. (2004) for instance, fitted separate multivariate time-series models to the crime trends in each project area in their study, with appropriate specifications for the particular properties of the time series in each project (capturing heterogeneous local patterning of crime), and then estimated unique project effects (measured by the Intensity of Project Action variable), net of ex ante projections of the crime trend due to other influences. Ekblom et al. (1996) carried out a highly-sophisticated study to adjust observed data to account for a large range of selection biases and heterogeneous effects that could not by controlled by research design. In conclusion, what may be needed to advance understanding of the impact of community crime prevention are inferential statistical models that seek to estimate selection bias and heterogeneity – properties fundamentally affecting evaluability – using counter-factual causal reasoning to underpin micro-econometric analysis (Figure 7).

114

Evaluation_Safety.book Page 115 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

Figure 6: Micro-Econometric Experiment

Figure 7: The Inferential Model

115

Evaluation_Safety.book Page 116 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

References BENNETT T., 1990, Evaluating Neighbourhood Watch, Aldershot, Gower. BENNETT T., 1991, The effectiveness of a police-initiated fear-reducing strategy, British Journal of Criminology, 31, 1-14. BENNETT T., 1996, What’s new in evaluation research? A note on the Pawson and Tilley article, British Journal of Criminology, 36, 4, 567-573. BOWLES R., PRADIPTYO R., 2004, Reducing Burglary Initiative: an analysis of costs, benefits and cost effectiveness, London, Home Office, [http:// rds.homeoffice.gov.uk/rds/pdfs04/rdsolr4304.pdf]. BRANTINGHAM P.J., FAUST F.L., 1976, A conceptual model of crime prevention, Crime and Delinquency, 2, 3, 284-296, [reprinted in HOPE T., (Ed.), 2000, Perpectives on crime reduction, Aldershot, Ashgate, 315]. CAMPBELL, D.T., 1969, Reforms as Experiments, American Psychologist 24, 4, 409-429 (Reprinted as CAMPBELL, D.T. 1978 Reforms as Experiments, in BYNNER J., STRIBLEY K.M., (Eds), Social Research: Principles and Procedures, Oxford, Oxford University Press. CONNELL J. P., KUBISCH A. C., 1998, Applying a theory of change approach to the evaluation of comprehensive community initiatives: progress, prospects and problems in FULBRIGHT-ANDERSON K., KUBISCH A.C., CONNELL J. P., New Approaches to Evaluating Community Initiative, Volume 2: Theory, Measurement, and Analysis, Washington DC, The Aspen Institute. COOK T.D., CAMPBELL D.T., 1979, Quasi-Experimentation: Design and Analysis Issues for Field Settings, Boston, MA, Houghton Mifflin. CRAWLEY P., HOPE T., 2003, Measuring the outputs of crime reduction projects, Community Safety Journal, 2, 3, 4-12. CROW I., FRANCE A., HACKING S. HART M., 2004, Does Communities that Care Work? York, The Joseph Rowntree Foundation. EKBLOM P., LAW H., SUTTON M. (with CRISP P., WIGGINS R.), 1996, Safer Cities and Domestic burglary, Home Office Research Study 164, London, Home Office Research and Statistics Directorate, [http:// www.homeoffice.gov.uk/rds/pdfs/hors164.pdf]. FARRELL G., 2005, Progress and prospects in the prevention of repeat victimisation in N. TILLEY (Ed.) Handbook of Crime Prevention and Community Safety, Cullompton, Devon, Willan. FARRINGTON D. P., 1997, Evaluating a Community Crime Prevention Program, Evaluation, 3, 2, 157-173. FARRINGTON D. P., 1998, Evaluating 'Communities That Care': Realistic Scientific Considerations’, Evaluation, 4, 2, 204-210.

116

Evaluation_Safety.book Page 117 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

FARRINGTON D. P., 2003, Methodological quality standards for evaluation research, The Annals of the American Academy of Political and Social Science, 587, 49-68. FORRESTER D., CHATTERTON M., PEASE K., 1988, The Kirkholt Burglary Prevention Project, Rochdale, Crime Prevention Unit Paper 13. London, Home Office, [http://www.homeoffice.gov.uk/rds/prgpdfs/fcpu13.pdf]. FORRESTER D., FRENZ S., O’CONNELL M., PEASE K., 1990, The Kirkholt Burglary Prevention Project, Phase II, Crime Prevention Unit Paper 23. London, Home Office, [http://www.homeoffice.gov.uk/rds/prgpdfs/ fcpu23.pdf]. FOSTER J., HOPE T., 1993, Housing, Community and Crime: the impact of the Priority Estates Project, Home Office Research Study No. 131, London, HMSO, [http://rds.homeoffice.gov.uk/rds/pdfs05/hors131.pdf]. GILL, M., SPRIGGS A., 2005, Assessing the impact of CCTV, Home Office Research Study 292, London, Home Office Research, Development and Statistics Directorate, [http://www.homeoffice.gov.uk/rds/pdfs05/ hors292.pdf]. GLADSTONE F.J., 1980, Coordinating Crime prevention efforts, Home Office Research Study 62, London, Home Office Research, Development and Statistics Directorate, [http://www.homeoffice.gov.uk/rds/pdfs05/ hors62.pdf]. GOLDBLATT P., LEWIS C., (Eds.), 1998, Reducing Offending: an assessment of research evidence on ways of dealing with offending behaviour Home Office Research Study 187. London, Home Office, 51-62, [http:// rds.homeoffice.gov.uk/rds/pdfs/hors187.pdf]. HECKMAN J., 2001, Microdata, heterogeneity, and the evaluation of public policy: Nobel Lecture, Journal of Political Economy, 109, 3, 673-748. HECKMAN J., SMITH J.A., 1995, Assessing the case for social experiments, The Journal of Economic Perspectives, 9, 85-110. HERNÁN M.A., ROBINS J.M., 2006, Estimating causal effects from epidemiological data, Journal of Epidemiology and Community Health, 60, 7, 578-586. HOLLIN C. R., 2008, Offending behaviour programmes: Does only randomization glister? Criminology and Criminal Justice, 8, 1, 89-106. HOPE T., 1985, Implementing Crime Prevention Measures, Home Office Research Study No. 86, London, HMSO, [http:// www.homeoffice.gov.uk/rds/pdfs05/hors86.pdf]. HOPE T., 1995, Community crime prevention in TONRY M., FARRINGTON D.P., (Eds), Building a Safer Society: strategic approaches to crime prevention, Chicago, University of Chicago Press, Crime and Justice: a review of research, 19, 21-89.

117

Evaluation_Safety.book Page 118 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

HOPE T., 1998, Community crime prevention in NUTTALL C., GOLDBLATT P., LEWIS C., (Eds.), Reducing Offending: an assessment of research evidence on ways of dealing with offending behaviour, Home Office Research Study 187, London, Home Office, 51-62, [http:// rds.homeoffice.gov.uk/rds/pdfs/hors187.pdf]. HOPE T. 2002a, The road taken: evaluation, replication and crime reduction in HUGHES G., MCLAUGHLIN E., MUNCIE J., (Eds.), Crime Prevention and Community Safety, London, Sage Publications, 37-57. HOPE T., 2002b, La riduzione della criminalità, la sicurezza locale e la nuova filosofia del management pubblico, Dei Delitti e delle Pene, 9, 12-3 (seconda serie), 207-229. HOPE T., 2004, Pretend it works: evidence and governance in the evaluation of the Reducing Burglary Initiative. Criminology and Criminal Justice, 4, 3, 287-308. HOPE T., 2005a, The new local governance of community safety in England and Wales. Canadian Journal of Criminology and Criminal Justice, 47, 2, 367-387 HOPE T., 2005b, Sustainability in crime prevention: a nautical tale, 10th Anniversary Colloquium international Centre for the Prevention of Crime, Paris, 30 November-2 December, 2004. International Observer, Edition 36, March 2005, Montreal, International Centre for the Prevention of Crime, [http://www.crime-prevention-intl.org/ io_view.php?io_id=117]. HOPE T., 2005c, Pretend it doesn't work: the ‘anti-social’ bias in the Maryland Scientific Methods Scale, European Journal on Criminal Policy and Research, 11, 3-4, 275-296. HOPE T., 2006, Things can only get better, Criminal Justice Matters, No. 62 Winter 2005/06, [http://www.crimeandjustice.org.uk/opus171/cjm62hope.pdf], (Reprinted in House of Commons Science and Technology Committee Scientific Advice, Risk and Evidence Based Policy Making, Seventh Report of Session 2005-06, Volume II Oral and Written Evidence, Ev 145. The House of Commons, HC 900-II 8 November 2006). HOPE T., 2007, The Distribution of Household Property Crime Victimization: Insights from the British Crime Survey in HOUGH M., MAXFIELD M., (Eds.), Surveying Crime in the 21st Century, Crime Prevention Studies Vol. 22, Monsey, NY, Criminal Justice Press and Collumpton, Devon, Willan HOPE T., 2008a, A firing squad to shoot the messenger: Home Office peer review of research in MACMAHON W. (Ed.). Critical Thinking about the Uses of Research. Evidence Based Policy Series 1, London, Centre for

118

Evaluation_Safety.book Page 119 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

Crime and Justice Studies, Kings College London, [www.crimeandjustice.org]. HOPE, T., 2008b, The first casualty: evidence and governance in a war against crime in CARLEN P., (Ed), Imaginary Penalities, Cullompton, Devon, Willan. HOPE T., BRYAN J., CRAWLEY E., CRAWLEY P., RUSSELL N., TRICKETT A., 2004, Strategic Development Projects in the Yorkshire and the Humber, East Midlands and Eastern Regions, Home Office Online Report 41/04, London, Home Office, [http://rds.homeoffice.gov.uk/rds/pdfs04/ rdsolr4104.pdf]. HOPE T., KARSTEDT S., 2003, Towards a new social crime prevention in KURY H., OBERGFELL-FUCHS J., (Eds.), Crime Prevention: New Approaches, Mainz, Weisse Ring Verlag, 461-489, [http:// www.pbni.org.uk/bsthopeskarstedt.pdf]. HOPE T., MURPHY D.J.I., 1983, Problems of implementing crime prevention: the experience of a demonstration project, The Howard Journal, 22, 1-3, 38-50. HOPE T., SPARKS R., 2000, For a sociological theory of situations (or how useful is pragmatic criminology?), in VON HIRSCH A., GARLAND D., WAKEFIELD A., (Eds.), Situational Crime Prevention: ethics and social context, Oxford, Hart Publishing. HOPE T., TRICKETT A. 2004, Angst Essen Seele Auf…but it keeps away the burglars! private security, neighbourhood watch and the social reaction to crime, Kölner Zeitschrift für Soziologie und Sozialpsychologie, Sonderheft 43, 441-468. HIRSCHFIELD A., 2004, The impact of the Reducing Burglary Initiative in the North of England, Home Office Online Report 40/04, [http:// rds.homeoffice.gov.uk/rds/pdfs04/rdsolr4004.pdf]. JOSEPH ROWNTREE FOUNDATION, 2004, The Evaluation of Three ‘Communities that Care’ Demonstration Projects, Findings, York, The Joseph Rowntree Foundation. JUDD C.M., KENNY D.A., 1981, Estimating the Effect of Social Interventions, Cambridge, Cambridge University Press. KNOX J., PEMBERTON P. (PA CONSULTING GROUP), WILES P., 2000, Partnerships in Community Safety, London, Department of the Environment, Transport and the Regions. KODZ J., PEASE K. STEPHENS M., 2004, Explanatory note for online reports OLR 40/04 to OLR 43/04, Home Office Online Report 39/04, [http:// rds.homeoffice.gov.uk/rds/pdfs04/rdsolr3904.pdf]. MCLENNAN D., WHITWORTH A., 2008, Displacement of Crime or Diffusion of Benefit: evidence from the New Deal for Communities Programme,

119

Evaluation_Safety.book Page 120 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

Wetherby, W. Yorks, Communities and Local Government Publications. MAGUIRE M., 2004, The Crime reduction programme in England and Wales: reflections on the vision and reality, Criminology and Criminal Justice, 4, 3, 213-237. MILLIE A., M. HOUGH M., 2004, Assessing the impact of the Reducing Burglary Initiative in southern England and Wales. Home Office Online Report 42/04, [http://rds.homeoffice.gov.uk/rds/pdfs04/rdsolr4204.pdf]. O’MALLEY P., 2000, Risk, crime and prudentialism revisited in K. STENSON K., SULLIVAN R., (Eds.), Crime, Risk and Justice: the politics of crime control in neo-liberal societies, Cullompton, Devon, Willan. PAWSON R., TILLEY N., 1994, What works in evaluation research?, British Journal of Criminology, 34, 2, 291-306. PAWSON R., TILLEY N., 1996, What’s crucial in evaluation research: A reply to Bennett, British Journal of Criminology, 36, 4, 574-578. PAWSON R., TILLEY N., 1997, Realistic Evaluation, London, Sage Publications. PAWSON R., TILLEY N., 1998a, Caring Communities, Paradigm Polemics, Design Debates, Evaluation, 4, 1, 73-90. PAWSON R., TILLEY N., 1998b, Cook-Book Methods and Disastrous Recipes: A Rejoinder to Farrington, Evaluation, 4, 2, 211-213. QUINTON P., MORRIS J., 2008, Neighbourhood policing: the impact of piloting and early national implementation, Home Office Online Report 01/08, [http://rds.homeoffice.gov.uk/rds/pdfs08/rdsolr0108.pdf]. SHERMAN L.W., FARRINGTON D. P., WELSH B. C., MACKENZIE D. L., (Eds.), 2002, Evidence-based crime prevention, London/New York, Routledge. SHERMAN L.W., GOTTFREDESON D.C., MACKENZIE D.L., ECK J., REUTER P., BUSHWAY S., 1997, Preventing Crime: What Works, What Doesn’t, What’s Promising, Washington, D.C. US Office of Justice Programs, [http:// www.ncjrs.gov/works/]. SHERMAN L.W., GOTTFREDSON D., MACKENZIE D., ECK J., REUTER P., BUSHWAY S., 1998, Preventing Crime: What Works, What Doesn't, What's Promising. A Report to the United States Congress, Washington, National Institute of Justice Research in Brief. STC, 2006, Oral Evidence (Professor Tim Hope), Wednesday 24 May 2006. House of Commons Science and Technology Committee Scientific Advice, Risk and Evidence Based Policy Making, Seventh Report of Session 2005-06, Volume II Oral and Written Evidence, Ev 38. The House of Commons, HC 900-II 8 November 2006. TILLEY N., 1993, After Kirkholt – Theory, Method and Results of Replication Evaluations, Crime Prevention Unit Series Paper No. 47. London, Home Office, [http://rds.homeoffice.gov.uk/rds/prgpdfs/fcpu47.pdf]. 120

Evaluation_Safety.book Page 121 Tuesday, May 26, 2009 12:46 PM

Evaluation of Safety and Crime Prevention Policies in England and Wales

TILLEY N., LAYCOCK G., 2000, Joining up research, policy and practice about crime, Policy Studies, 21, 3, 213-227. TILLEY N., WEBB, J., 1994, Burglary Reduction: Findings from Safer Cities Schemes, Crime Prevention Unit Series, Paper No.51. London, Home Office, [http://rds.homeoffice.gov.uk/rds/prgpdfs/cpu51bf.pdf]. TUFFIN R., MORRIS J., POOLE A., 2006, An evaluation of the impact of the National Reassurance Policing Programme, Home Office Research Study 296. Home Office Research, Development and Statistics Directorate, [http://rds.homeoffice.gov.uk/rds/pdfs06/hors296.pdf].

121

Evaluation_Safety.book Page 122 Tuesday, May 26, 2009 12:46 PM

Evaluation_Safety.book Page 123 Tuesday, May 26, 2009 12:46 PM

On the Authors

Tim Hope is Professor of Criminology at Keele University, UK. He has been Senior Visiting Research Fellow, Scottish Centre for Crime and Justice Research, University of Edinburgh, and has held positions at the Universities of Manchester, Missouri-St. Louis USA, and the Home Office Research and Planning Unit (1974-1991). He was Director of the Economic and Social Research Council’s / Crime and Social Order Research Programme (1993-1998). For the past thirty-years he has been involved in the analysis, design and direction of evaluation research in community crime prevention. His recent articles on policy evaluation include: 앫 The first casualty: evidence and governance in a war against crime, in Carlen P., (Ed), Imaginary Penalities, Cullompton, Devon, Willan, 2008. 앫 Pretend it doesn’t work: the ‘anti-social’ bias in the Maryland Scientific Methods Scale, European Journal on Criminal Policy and Research, 2005, 11, 3-4, 275-296. 앫 Pretend it works: evidence and governance in the evaluation of the Reducing Burglary Initiative, Criminology and Criminal Justice, 2004, 4, 3, 287-308. He also gave oral and written evidence to the House of Commons Science and Technology Select Committee inquiry into /Scientific Advice, Risk and Evidence Based Policy Making/ (2006). Philippe Robert is a sociologist, directeur de recherches émérite at the Centre national de la recherche scientifique (CNRS). He is the founder and former head of the Centre de recherches sociologiques sur le droit et les institutions pénales (CESDIP), the main French research centre on crime. He has also founded the Groupe européen de recherches sur les normativités (GERN), an European scientific network. Within the Coordination Action CRIMPREV, he coordinated the workshop of which this volume is one of the outcomes. His current fields of research are sociological theory of crime, measuring crime, victimisation and insecurity surveys. Among his recent publications: 앫 Sociologia do crime, Petropolis, Vozes (www.editoravozes.com.br), 2007. 앫 Sociologie du crime, Paris, La Découverte (www.editionsladecouverte.fr), Coll. Repères, 2005. 앫 Bürger, Kriminalität und Staat, Wiesbaden, VS Verlag für Sozialwissenschaften/GmbH (www.vs-verlag.de), 2005. 123

Evaluation_Safety.book Page 124 Tuesday, May 26, 2009 12:46 PM

Evaluating Safety and Crime Prevention Policies in Europe

Sybille Smeets is a researcher at the Centre of criminological researches and assistant Professor at the School of Criminology at the Université Libre de Bruxelles (ULB). Her research areas include police and policing, local prevention policies and links between security policies and Welfare State. Her latest books as co-editor are: 앫 Polices locales et autorités administratives. Je t’aime, moi non plus, Bruxelles, Politeia, 2008. 앫 La fraude sociale. Une nouvelle priorité de politique criminelle?, Bruxelles, Bruylant, 2009. She has also recently published (with C. Tange): 앫 Community Policing in Belgium: The Vicissitudes of the Development of a Police Model », in Wisler D., Onwudiwe I., (Eds)., Patterns of Community Policing: An International Comparative Perspective, London, CRC Press, 2009, 125-148. Carrol Tange is a researcher at the Centre of criminological researches (ULB), assistant at the Facultés universitaires Saint-Louis (FUSL) and substitute professor at the Université de Mons-Hainaut (UMH). His research areas include police policies, practices and management theories. His latest books as co-editor are: 앫 La police au Québec. L’art du réformisme pragmatique, Bruxelles, Politeia, 2007. 앫 Out of the box. Un livre pour colorer et penser l’interculturalité et la police : La diversité dans la diversité, Bruxelles, Politeia, 2008. 앫 Insights on Police. Quarter of a Century Research on Police in Europe and the Anglo-Saxon world, Brussels, Bruylant, 2009. Karin Wittebrood is a senior researcher at the Netherlands Institute of Social Research (Sociaal en Cultureel Planbureau, SCP). Her interests are on criminal victimisation and fear of crime, the impact of social context on these issues and the effectiveness of policy measures to reduce (fear of) crime. Recent publications have appeared in The British Journal of Criminology, European Sociological Review, Homicide Studies and Social Problems. Anne Wyvekens is a researcher at the Centre d’études et de recherches des sciences administratives et politiques (CERSA), a research centre affiliated to the French Centre national de la recherche scientifique (CNRS) and the Université Panthéon-Assas, and invited professor at the Facultés universitaires Saint Louis (FUSL, Brussels).

124

Evaluation_Safety.book Page 125 Tuesday, May 26, 2009 12:46 PM

On the Authors

She worked as a scientific expert for the Council of Europe’ Committee on Partnership in the prevention of crime and contributed to it subsequent publication (A Partnership Approach to Crime Prevention, Strasbourg, Council of Europe, 2004). From 2002 to 2005, she was director of the Research at the Institut des hautes études de la sécurité intérieure (IHESI, French Home Department). She is also a member of the scientific committee of the International Centre fort he Prevention of Crime (ICPC) and works with the European Forum for Urban Safety (EFUS). Her research fields are criminal justice, crime prevention, local safety policies, juvenile justice, urban policies in France as well as in a comparative world (Europe, USA). Her recent publications include: 앫 Espace public et sécurité, Paris, La Documentation française, 2006 (with Jacques Donzelot) 앫 La magistrature sociale, Paris, La Documentation française, 2004 앫 Faire social, la politique de la ville aux États-Unis et en France, Paris, Seuil, 2003 (with Jacques Donzelot and Catherine Mevel). Renée Zauberman is a senior researcher at the Centre de recherches sociologiques sur le droit et les institutions pénales, a research centre affiliated to the French Centre National de la Recherche scientifique, the Université de Versailles Saint Quentin (UVSQ) and the French Department of Justice. Her interest lies is norms and deviance, especially criminal justice and crime. While she has worked on the French Gendarmerie, her current focus is on victimisation surveys. She has published: 앫 Zauberman R. Ed., Victimisation and Insecurity in Europe ; A Review of Surveys and Their Use, Brussels, VUBPress, 2008. 앫 L’évolution de la délinquance d’après les enquêtes de victimation. France, 1984-2005, Déviance & Société, 2008, 32, 4, 435-472 (with Ph. Robert, S. Névanen, E. Didier). 앫 Police, Minorities and the French Republican Ideal, Criminology, 2003, 41, 4, 1065-1100 (with R. Lévy).

125