Algorithmic Regulation: a critical regulation

Citation preview

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/318820255

Algorithmic regulation: A critical interrogation Article  in  Regulation & Governance · July 2017 DOI: 10.1111/rego.12158

CITATIONS

READS

17

1,268

1 author: Karen Yeung University of Birmingham 32 PUBLICATIONS   513 CITATIONS    SEE PROFILE

All content following this page was uploaded by Karen Yeung on 09 February 2018. The user has requested enhancement of the downloaded file.

(Yeung Algorithmic Regulation 2017 accepted.doc)

The published version of this paper will appear in a forthcoming issue of Regulation & Governance available at http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1748-5991

Algorithmic Regulation: A Critical Interrogation Karen Yeung

Professor of Law, Centre for Technology, Ethics, Law & Society (TELOS) King’s College London

This paper has been accepted for publication by Regulation & Governance. Please do not distribute without the author’s permission. Abstract: Innovations in networked digital communications technologies, including the rise of ‘Big Data’, ubiquitous computing and cloud storage systems, may be giving rise to a new system of social ordering known as algorithmic regulation. Algorithmic regulation refers to decision-making systems that regulate a domain of activity in order to manage risk or alter behaviour through continual computational generation of knowledge by systematically collecting data (in real time on a continuous basis) emitted directly from numerous dynamic components pertaining to the regulated environment in order to identify and, if necessary, automatically refine (or prompt refinement of) the system’s operations to attain a pre-specified goal. It provides a descriptive analysis of algorithmic regulation, classifying these decisionmaking systems as either reactive or pre-emptive, and offers a taxonomy that identifies 8 different forms of algorithmic regulation based on their configuration at each of the three stages of the cybernetic process: notably, at the level of standard setting (adaptive vs fixed behavioural standards); information-gathering and monitoring (historic data vs predictions based on inferred data) and at the level of sanction and behavioural change (automatic execution vs recommender systems). It maps the contours of several emerging debates surrounding algorithmic regulation, drawing upon insights from regulatory governance studies, legal critiques , surveillance studies and critical data studies to highlight various concerns about the legitimacy of algorithmic regulation

Keywords: big data, algorithms, surveillance, enforcement, automation



1

Algorithmic Regulation : A Critical Examination 1.

Introduction

“It’s time for government to enter the age of big data. Algorithmic regulation is an idea whose time has come” (Tim O’Reilly, CEO of O’Reilly Media Inc). A so-called Big Data revolution is currently underway, which many claim will prove as disruptive to society in the 21st century as Henry Ford’s system of mass production in the late 19th century (boyd and Crawford 2012). Although ‘Big Data’ has been variously defined, I use the term to refer to the socio-technical ensemble which utilises a methodological technique that combines a technology (constituted by a configuration of information-processing hardware and software that can sift and sort vast quantities of data in very short times) with a process (through which algorithmic processes are applied to mine a large volume of digital data to find patterns and correlations within that data, distilling the patterns into predictive analytics, and applying the analytics to new data) (Cohen 2012: 1919). The excitement surrounding Big Data is rooted in its capacity to identify patterns and correlations that could not be detected by human cognition, converting massive volumes of data (often in unstructured form) into a particular, highly data-intensive form of knowledge, and thus creating a new mode of knowledge production (Cohen 2012: 1919). Industries, academics and governments are enthusiastically embracing these technologies, all seeking to harness their tremendous potential to enhance the quality and efficiency of many activities, including the task of regulation, which this paper interrogates by critically examining the phenomenon of ‘algorithmic regulation’. It draws upon selective insights from legal and social scientific literature, highlighting emerging critiques of algorithmic power and the rise of automated data-driven systems to inform decision-making and regulate behaviour. My primary aim is to map the contours of emerging debates, raising questions for further research rather offering definitive answers, proceeding in four parts. Part I offers a working definition of algorithmic regulation. Part II then constructs a taxonomy of algorithmic regulatory systems based on their configuration at each of the three stages of the cybernetic process: notably, at the level of standard setting (adaptive vs simple, fixed behavioural standards); information-gathering and monitoring (historic data vs predictions based on inferred



2

data), and at the level of sanction and behavioural change (automatic execution vs recommender systems). Parts III, IV and V provide a critical analysis of algorithmic regulation, identifying concerns about its legitimacy drawn selectively from several strands of academic literature, including regulatory governance and public administration, legal scholarship, surveillance studies and critical data studies. Part VI concludes, sketching the contours of a broad research agenda anchored within legal and regulatory governance scholarship. 2.

The mechanisms and forms of algorithmic regulation

2.1

What is algorithmic regulation?

Although Silicon Valley entrepreneur Tim O’Reilly exhorts governments to embrace algorithmic regulation to solve policy problems he does not define algorithmic regulation, but merely points to various technological systems1 which he claims share four features: (a) a deep understanding of the desired outcome; (b) real-time measurement to determine if that outcome is being achieved; (c) algorithms (i.e. a set of rules) that make adjustments based on new data; and (d) periodic, deeper analysis of whether the algorithms themselves are correct and performing as expected (O’Reilly 2013). Because greater precision and rigour is required for critical analysis, I begin by offering a definition of algorithmic regulation, exploring what it means to describe something as ‘algorithmic’, and then explaining how I will understand the term ‘regulation’. In their broadest sense, algorithms are encoded procedures for solving a problem by transforming input data into a desired output (Gillespie 2013; 2014). Although algorithms need not be implemented in software, computers are fundamentally algorithm machines, designed to store and read data, apply mathematical procedures to data in a controlled fashion, and offer new information as the output. Even when confined to software, the term ‘algorithm’ may be variously understood. Software engineers are likely to adopt a technical understanding of algorithms , referring to the logical series of steps for organising and acting on a body of data to achieve a desired outcome quickly which

1

He refers to motor vehicle fuel emissions systems, airline automatic pilot systems, credit card fraud detection systems, drug dosage monitoring by medical professionals, internet spam filters and general internet search engines: O’Reilly (2013).



3

occurs after the generation of a ‘model’, i.e. the formalisation of the problem and the goal in computational terms (Gillespie 2013; 2014; Dourish 2016). But social scientists typically use the term as an adjective to describe the sociotechnical assemblage which includes, not just algorithms, but also the computational networks in which they function, the people who design and operate them, the data (and users) on which they act, and the institutions that provide these services, all connected to a broader social endeavour and constituting part of a family of authoritative systems for knowledge production. Accordingly, Gillespie suggests that, when describing something as ‘algorithmic’, our concern is with the insertion of procedure that is produced by, or related to, a socio-technical information system that is intended by its designers to be functionally and ideologically committed to the computational generation of knowledge. For him, ‘what is central is the commitment to procedure, and the way procedure distances its human operators from both the point of contact with others and the mantle of responsibility for the intervention they make’ (Gillespie 2014). Although computational algorithms include those which encode simple mathematical functions, the excitement surrounding Big Data is largely attributable to sophisticated machine learning algorithms , fed by massive (and often unstructured) data sets, that operate computationally and depart from traditional techniques of statistical modelling (Dourish 2016: 7). Traditional statistical modelling requires the analyst to specify a mathematical function containing selected explanatory variables and, through regression analysis, enables the identification of the goodness of fit between the data and these analytic choices. In contrast, machine learning does not require a priori specification of functional relationships between variables. Rather, the algorithms operate by mining the data using various techniques2 to identify patterns and correlations between the data and which are used to establish a working model of relationships between inputs and outputs. This model is gradually improved by iterative ‘learning’ that is, by testing its predictions and correcting them when wrong, until it identifies something like what is understood in conventional statistics as a ‘line of best fit’ to generate a model that provides the strongest predictive relationship between inputs and outputs.3 Accordingly,

2

Five well-used techniques are logistic regression models, the naïve Bayes classifier, knearest neighbors, decision trees and neural networks, all of which exemplify predictive modelling: Mackenzie (2015) 432-433. 3 Machine learning algorithms can be broadly split into three categories based on how they learn. Supervised learning requires a training data set with labeled data, or data with a known



4

this methodological approach is sometimes described as ‘letting the data speak’4 (MayerSchonberger and Cukier 2013: 6). While conventional statistical regression models worked with 10 or so different variables (such as gender, age, income, occupation, educational level, income and so forth) and perhaps sample sizes of thousands, machine learning algorithms that drive the kind of predictive analytic tools that are now commonly in use are designed to work with hundreds (and sometimes tens of thousands) of variables (‘features’) and sample sizes of millions or billions (Mackenzie 2015: 434). Algorithmic decision-making refers to the use of algorithmically generated knowledge systems to execute or inform decisions, and which can vary widely in simplicity and sophistication. Algorithmic regulation refers to regulatory governance systems that utilise algorithmic decision making. Although the scope and meaning of the term ‘regulation’ and ‘regulatory governance’ is contested (Baldwin et al 2010), I adopt the definition offered by leading regulatory governance scholar Julia Black, who defines regulation (or regulatory governance) as intentional attempts to manage risk or alter behaviour in order to achieve some pre-specified goal (Black 2014). Several features of this understanding of regulation are worth highlighting. First, although regulation is

output value. Unsupervised learning techniques do not use a training set, but find patterns or structure in the data by themselves. Semi-supervised learning uses mainly unlabelled and a small amount of labelled input data. Using a small amount of labelled data can greatly increase the efficiency of unsupervised learning tasks. The model must learn the structure to organize the data as well as make predictions (NESTA 2015: 5). 4 There is a growing literature in ‘Critical Data Studies’ (or ‘Critical Algorithm Studies’) which seeks to explore data as situated in complex ‘data assemblages’ of action (Kitchen 2014b: 24-26), referring to the vast systems, comprised not just of database infrastructures, but also the ‘technological, political, social and economic apparatuses that frames their nature, operation and work’, including processes of data collection and categorization to its subsequent cleaning, storing, processing, dissemination and application (Kitchen et al 2015). A growing body of research examines the labour and political economies entailed in the reproduction of these assemblages, using a wide range of disciplinary lenses including STS (Ziewieke 2016; Beer 2017; MacKenzie 2015; Cheney-Lippold 2011) focusing on data from a variety of sources including meteorological data, data produced by for-profit education companies, financial trading data and biomedical data. This literature exposes the fallacy of understanding data as an objective set of facts that exist prior to ideology, politics or interpretation by seeking to understand data as situated in socio-technical systems that surround its production, processing, storing, sharing, analysis and reuse, thereby demonstrating that the production of data assemblages is not a neutral, technical process, but a normative, political and ethical one that is contingent and often contested, with consequences for subsequent analysis, interpretation and action (Kitchen 2014a). An illustration of the wide range of disciplinary perspectives, questions and approaches emerging within this field can be found in Big Data & Society (2016), Special Issue on Critical Data Studies available at http://bds.sagepub.com/content/critical-data-studies (accessed 11 November 2016).



5

widely regarded as a critical task of governments, regulation is also pursued by non-state actors and entities (Black 2008). Just as a public transport authority may regulate vehicle movement to optimise traffic flow, likewise a social media company such as Facebook might regulate the posting and viewing behaviour of users to optimise its financial returns. Secondly, the size of the regulated ‘population’ is highly variable. It may refer to the intentional actions of one person who adopts some system or strategy to regulate some aspect of her own behaviour (such as an individual who uses a fitness tracking device to help her ensure that she attains a minimum level of daily physical activity) through to regulatory systems that seek to direct and influence the behaviour of a large number of people or entities, such as algorithmic systems employed by digital car-sharing platforms, Uber, to enable drivers to offer motor vehicle transport services to individuals at a pre-specified fee without having had any previous relationship. Thirdly, because regulation is above all an intentional activity directed at achieving a pre-specified goal, any regulatory system must have some kind of system ‘director’ (or ‘regulator’) to determine the overarching goal of the regulatory system. Accordingly, I refer to algorithmic regulation as decision-making systems that regulate a domain of activity in order to manage risk or alter behaviour through continual computational generation of knowledge from data emitted and directly collected (in real time on a continuous basis) from numerous dynamic components pertaining to the regulated environment in order to identify and, if necessary, automatically refine (or prompt refinement of) the system’s operations to attain a pre-specified goal. 2.2

Forms of Algorithmic Regulation: A Taxonomy

Algorithmic regulation has antecedents in the interdisciplinary science of cybernetics that emerged in the aftermath of World War II. Cybernetic analysis sought to move away from linear understanding of cause and effect and towards investigation of control through circular causality, or feedback (Medina 2015). The logic underpinning algorithmic regulation, and the ‘smartification’ of everyday life which it makes possible, rests on the continuous collection and analysis of primary data combined with metadata, which logs the frequency, time and duration of device usage and which, via direct machine-to-machine communication via digital networks, allows the combined data to be algorithmically mined in order to trigger an automated response (Morozov 2014).



6

By understanding regulation as a cybernetic process involving the three core components of any control system – i.e. ways of setting standards, goals or targets (‘standardsetting’); ways of gathering information (‘information-gathering’) and ways of enforcing those standards once deviation is identified in order to change behaviour so that it meets the requisite standards (‘enforcement and behaviour modification’), various forms of algorithmic regulation can be identified (Hood et al 2001: 23; Morgan and Yeung 2007:3 ). My taxonomy identifies two alternative configurations for each component, thereby generating a total of eight different forms (see Table 1). First, at the level of standard setting, the behavioural norm which the system enforces may be either a simple, fixed (yet reprogrammable) standard of behaviour. This is the most basic form of algorithmic intervention, exemplified in the use of password protection systems to authorise access to digital content. Alternatively, the behavioural standard may be adaptive, to facilitate the attainment of whatever fixed, overarching (yet reprogrammable) system goal the regulatory system is designed to optimise in order to produce system stability. These latter systems are often described as ‘complex’ or ‘intelligent’, such as intelligent transportation systems that effectively teach themselves how to identify the most reliable predictor of traffic flow through machine learning processes that utilise trial and error applied to continuously updated real-time traffic data (Lv et al 2015). Although these systems allow behavioural variation, for example, in vehicle speed limits and/or the duration, timing and frequency of traffic light cycles, depending upon traffic volume and distribution, the overarching objective is pre-specified and fixed by the system director: put simply, to optimise traffic flow within the system.5 Second, at the level of information gathering and monitoring, the system may operate on a reactive basis, configured automatically to mine historic performance data in real-time to detect violation. Simple reactive systems include automated vehicle speeding detection systems that utilise speed cameras to provide real-time identification of vehicles that exceed prescribed speed limits, while complex reactive systems include credit card fraud detection systems that utilise machine learning techniques to profile the

5

Algorithmic systems of this kind also underpin growing labour market practices including the use of ‘zero hour contracts’ which subject workers to variable scheduling, focusing paid work hours to times of high demand thus shifting the risk of changing demand onto workers and increasing work intensity (Wood 2016). Similarly, algorithmic workplace performance management techniques rely on microlevel surveillance of call centre workers to provide feedback to employers and employees aimed at optimising worker productivity (Kuchler 2014; Edwards and Edwards 2016).



7

spending patterns of credit card holders, aimed at detecting suspicious transactions when they occur, immediately alerting the credit provider and/or card-holder to take action. Alternatively, algorithmic systems may be configured to detect violations on a pre-emptive basis, applying machine learning algorithms to historic data to infer and thereby predict future behaviour. Simple predictive systems include digital text autocomplete systems while complex, pre-emptive systems make possible new forms of personalised pricing, applying machine learning algorithms to data collected from on-line tracking and measurement of on-line user behaviour at a highly granular level to generate consumer profiles, varying the price of goods offered to individuals on-line, based on algorithmic evaluations of the user’s willingness and ability to pay (Miller 2014). Third, at the level of enforcement and behaviour modification, the system may automatically administer a specified sanction or decision without any need for human intervention beyond user input of relevant data (or data tokens), such as simple reactive systems that automatically block access to web-content if the user fails to enter an authorised password. These systems constitute a form of action-forcing (or coercive) design (Yeung and Dixon-Woods 2010) thus offering the promise of immediate ‘perfect enforcement’ (Zittrain 2009)6. These systems may also operate pre-emptively, based on algorithmically determined predictions of a candidate’s future behaviour, such as systems that automatically evaluate applications from individuals seeking access to services such as loan finance, insurance cover and employment opportunities (O’Neil 2016). Although a human operator might be authorised to review and override the automated decision at a later stage, automation is relied upon to make and implement a decision that has real, consequential effects for the individual.7 Alternatively, both simple and complex systems

6

Although the self-executing capacity of these systems holds considerable allure by rendering human enforcement agents redundant, the legitimacy of ‘perfect enforcement’ has been questioned by cyberlawyer Jonathan Zittrain who highlights the dangers of smart devices (which he termed ‘tethered appliances’) emerging in an earlier internet age because they ‘invite regulatory intervention that disrupts a wise equilibrium that depends upon regulators acting with a light touch, as they traditionally have done within liberal societies’ (Zittrain 2009: 103). Moreover, the promise of ‘perfect’ enforcement is illusory, given the inevitable impossibility of defining ‘perfect’ standards that are capable of anticipating every single possible future event that may be of relevance to the operation of the regulatory system’s goals (Yeung 2008: 92-93). 7

It is the use of these kinds of algorithmic decision-making systems that has given rise to increasing concerns about errors in the underlying data, their application to particular individuals, and their potential discriminatory effects. Concerns of this kind have spawned a rising chorus of concern about the need for mechanisms that can secure algorithmic accountability, which are discussed more fully at section 5 below.



8

may be configured to provide automated ‘assistance’ or ‘recommendations’ to a human decision-maker, by prioritising candidates from within the larger regulated population. These ‘recommender systems’ are intended to direct or guide an individual’s decisionmaking processes in ways identified by the underlying software algorithm as optimal, offering prompts that focus a human user’s attention on a particular set of entities within the data set, with the human user retaining formal decision-making authority, exemplified by on-line shopping recommendation engines (Yeung 2017). Each of these forms of algorithmic regulation can be employed by state and non-state institutions. Some systems are applied to regulate the conduct of many millions of individuals, such as Facebook’s News Feed system (Luckerson 2015), while others may be limited to managing relationships within a small group. In particular, when used to manage the relationship between parties to a contract, they have been referred to by others as computer mediated contracts, referring to arrangements between contracting parties which harness the capacity of networked communication systems to undertake continuous, real-time digital monitoring of behaviour to detect, monitor and enforce performance of the terms of the contract, thereby overcoming a significant limitation of conventional contracts : the need for the principal to monitor the behaviour of the agent to guard against the agent’s temptation to ‘shirk’, that is, to act in a self-interested manner that is contrary to the interests of the principal (Williamson 1975). Hal Varian, Google’s Chief Economist, provides two examples of computer-mediated contracts that significantly reduce these costs for the principal: first, remote vehicle monitoring systems that verify whether driver behaviour conforms with the desired standard, thereby enabling car rental companies to continuously monitor and verify whether a driver is honouring his/her contractual obligation to operate the car in a safe manner. Secondly, these vehicle monitoring systems enable automated remote enforcement that allow a lender easily to repossess a car purchased by an individual on loan finance who fails to make two consecutive monthly repayments by automatically immobilising the car (Varian 2014). Table 1: A taxonomy of algorithmic regulatory systems

1.

Standard setting Fixed

2.

Fixed

3.

Fixed



Monitoring Real time reactive violation detection Real time reactive violation detection Pre-emptive

Enforcement/ Sanction Automated Recommender system Automated

Description Simple real-time sanction administration systems Simple real-time warning systems Simple pre-emptive sanction

9

4.

Fixed

5.

Adaptive

6.

Adaptive

7.

Adaptive

8.

Adaptive

violation prediction Pre-emptive violation prediction Real time reactive violation detection Real time reactive violation detection Pre-emptive violation prediction Pre-emptive violation prediction

Recommender system Automated Recommender system Automated Recommender system

administration systems Simple predictive recommender system Complex sanction administration systems Complex real-time prioritization systems Complex predictive sanctioning systems Complex predictive recommender systems

Taken together, these forms of algorithmic regulation draw upon a modality of control which regulatory governance scholars sometimes refer to as ‘architectural’ or ‘designbased’ techniques of control (Yeung 2008; Yeung 2015). On this view, algorithmic regulation is simply an application of the use of design for regulatory purposes, building on the logic of the waterlock, devised by the Ancient Egyptians over 3000 years ago to prevent unauthorised entry to a protected space without the need for human intervention (Beniger 1986). Although simple reactive algorithmic systems, such as digital password protection systems, are based on the same logic underpinning primitive control systems, they are of contemporary interest because digital sensor technology and storage capacity have now become so cheap that it has become practically feasible to utilise these systems in order to achieve regulatory policy goals (Rieder and Simon 2016). Hence, Proctor and Gamble’s smart soap dispenser, recently installed in some public conveniences in the Phillipines which operate by movement sensor technology which triggers an alarm when a user leaves the stall that stops only when the soap-dispensing button is pressed, is not especially novel as an automated control system. Yet these simple algorithmic systems are nevertheless worthy of critical attention because they have become attractive as tools for influencing the behaviour of multiple users, rather than being limited to conventional approach which, in the realm of public hygiene promotion, typically involves displaying a sign in public facilities exhorting users to wash their hands. Nevertheless, contemporary algorithmic regulation can be distinguished from their simple pre-digital antecedents in two respects. Firstly, complex forms of algorithmic regulation which are configured to optimise a fixed (but reprogrammable) overarching system goal while allowing variable adaptation of behavioural standards within the system by utilising machine learning techniques that operate on large multiple sources of data collected from ubiquitous digital sensors that continuously track behaviour, offer a



10

genuinely novel form of design-based control. Secondly, pre-emptive algorithmic systems are vastly more powerful than traditional forms of architectural regulation (such as the speed hump or the door lock, for example) because machine learning algorithms, networked communications infrastructure and cloud computing, make it possible to track and intervene in the behaviour of not just a single user, but also an entire population of users across a widely dispersed geographic area, while collecting and analysing population-wide data on an almost instantaneous basis to identify deviations from the system’s goal and, increasingly, to predict both individual and population-wide trends that can reveal, and automatically act upon, ‘hidden’ insight. It is the capacity of these systems to predict future action or behaviour based on the algorithmic identification of unexpected correlations within massive data sets that would not be detectable by human cognition (or even ordinary computing techniques) to generate ‘actionable insight’ that is widely regarded as the ‘Holy Grail’ of Big Data (Ekbia et al 2015). 3.

Understanding the Logic of Algorithmic Regulation as a Form of Social Ordering

Although my taxonomy identifies eight different forms of algorithmic systems, the following discussion broadly distinguishes reactive systems (which trigger an automated response based on algorithmic analysis of historic data) from pre-emptive systems (which act pre-emptively based on algorithmic assessment of historic data to infer predictions about future behaviour), drawing on various strands of regulatory governance literature and insights from social scientific to illuminate differences in their underlying logic. 3.1.

Reactive algorithmic systems

The logic underpinning reactive algorithmic systems can be examined by drawing on insights from so-called ‘tools of government’ literature within public administration (Hood and Margetts 1986) which interrogates the various instruments and techniques employed by public administrators to oversee and control a set of actors and their activities (Ogus 2004; Scott and Murray 2002). Two related approaches to regulatory control resonate with the motivating logic of algorithmic regulation: outcome-based regulation and performance-management systems:



11

a)

As a form of outcome-based regulation? O’Reilly’s enthusiasm for algorithmic

regulation appears to rest on his belief in the superiority of outcome-focused regulation, arguing that regulatory regimes should focus on the achievement of observable (and preferably measurable) substantive outcomes, rather than regulating the process by which the desired outcome is achieved (O’Reilly 2013). This claim resonates with a distinction drawn by regulatory governance scholars between output (or performance) standards, which require certain conditions of quality to be met at the point of supply but allows the supplier to choose how to meet those conditions and process (or input) standards, which either compel the supplier to employ certain processes or materials or prohibits the use of certain processes or materials (Ogus 1994: 151). This literature offers a more nuanced view of the benefits and shortcomings of outcome-based standards. From the perspective of the economic analysis of law, for example, the most important variables are the costs of being informed of the technological means of achieving regulatory goals compared with the administrative costs of formulating appropriate standards and monitoring compliance. Output standards are generally favoured because firms are free to determine how to meet regulatory goals, encouraging innovation in loss mitigation techniques in contrast to process-based standards, which are prone to obsolescence and can induce technological rigidity (Ogus 1994: 166). Yet others observe that, for many consumers, ‘process matters,’ with many willingly paying higher prices for products that conform to fair trade, fair labour or environmental process standards, even though their outputs may be indistinguishable from those produced in ways that do not conform with so-called ‘ethical’ or ‘socially responsible’ processes of production (Kysar 2004; O’Rourke 2003). Accordingly, the outcome-driven logic which O’Reilly identifies with algorithmic regulation is unlikely to be a superior across-the-board approach to regulation: a more contextual assessment is required to identify the regulatory form (whether algorithmic or otherwise) best suited to any particular set of circumstances. Moreover, algorithmic regulatory systems need not be configured to optimise substantive outcomes: system goals could also be configured to optimise procedural goals.8 In other words, O’Reilly’s assumption that algorithmic forms of regulation are necessarily directed towards the achievement of substantive outcomes is based on a particular set of normative assumptions, and it is these value-laden, political and

8



I am indebted to Henry Rothstein for this observation.

12

ideological dimensions that are highlighted by scholars who emphasise the value of process-based regulatory standards, a matter explored more fully in section 4.1 below. b)

As data-driven performance management: The logic underpinning reactive

algorithmic systems also strongly resembles performance management techniques9 prominent in recent contemporary public sector administration (Perrin 2015). Performance-based approaches that focus on results, rather than on governmental processes and activities, in order to drive up improvement in public sector administration, have been extensively studied and criticised. For example, in a study of ‘governance by targets’ in the English NHS adopted by the Blair administration from the late 1990s onwards, Hood and Bevan demonstrate that the theory of governance by targets requires two sets of heroic assumptions: of robust synecdoche and game proof design, which their study indicated were not justified. So although there were dramatic improvements in reported performance of NHS organisations, the extent to which the improvements were genuine or merely apparent due to gaming, misreporting or circumvention strategies are unknown (Bevan and Hood 2006). Similarly, Perrin observes that in practice, public sector management approaches that focus on measured performance outcomes have been deeply problematic, identifying significant shortcomings including: failing to provide a meaningful account of actual performance; the occurrence of ‘goal displacement’ in which an emphasis is placed on ‘making the numbers’ rather than larger programme objectives; imposing substantial pressure on managers to distort results (sometimes with devastating consequences); disguising significant in-group differences; inhibiting rather than improving performance; obstructing co-operative action with those outside the domain of measured performance; failure to provide meaningful guidance to decision-makers owing to their failure to identify the causal basis of performance outcomes and, ultimately, eroding confidence in government (Perrin 2015). These concerns resonate strongly with recent critiques of algorithmic tools and systems that have been used to ‘weed out’ teachers

9

Performance management can be understood as a form of Taylorism. As David Lyon observes, the turn to commercial surveillance by marketing professionals begins with the efforts of Alfred Sloan and the development of market research in the 1920s. What began in the workplace as ‘scientific management’ with the work of FW Taylor and others was slowly transposed into the sphere of consumption in practices of ‘social Taylorism’ (Webster and Robins 1986). Here, the focus was less on the worker, more on the household. Sloan worked for General Motors, and pioneered the attempt to use scientific management principles for commodity markets and consumer behavior (Lyon 2007).



13

and other individual employees algorithmically identified as ‘under-performing’, an evaluation often starkly at odds with qualitative evaluations of those same individuals by those who work closely with them (O’Neil 2016). Reactive systems of algorithmic regulation (that is, algorithmic forms 1, 2, 5 and 6 listed in Table 1) draw upon the combined lineage of outcome-based regulation and performance management but with three claimed advantages. First, by replacing the need for human oversight with ubiquitous, networked digital sensors, algorithmic systems enable the monitoring of performance against targets at greatly reduced labour cost and human effort. Secondly, these systems appear to provide objective, verifiable evidence because knowledge of system performance is based on data is collected directly from a multitude of behavioural sensors embedded into the environment, thereby holding out the prospect of ‘game proof’ design (Rieder and Simon 2016; Jurgenson 2014). Thirdly, these systems operate dynamically, continuously fed by real-time data, allowing almost immediate intervention to direct or constrain the targeted behaviour, and thereby avoiding problems arising from out-of-date performance data. These claims warrant further scrutiny, but before doing so, it is helpful to understand differences between the logic of reactive and pre-emptive algorithmic systems. In particular, although the dynamic nature of reactive systems enables immediate intervention as soon as a violation of the behavioural standard occurs (which offers many advantages relative to static regulatory forms that rely on intermittent, periodic monitoring), reactive algorithmic systems do not enable anticipatory intervention before a violation occurs. It is this capacity to intervene pre-emptively which helps explain much of the current attraction and excitement surrounding algorithmic decision-making systems, and to which I first turn. 3.2

Pre-emptive Algorithmic Systems

While the use of predictive algorithms for intelligence gathering, border control (Amoore 2013; Owen 2015) and the distribution of digital media content (Luckerson 2015) has attracted considerable public and scholarly attention, the use of predictive algorithms to inform the exercise of enforcement discretion for the purposes of detecting or preventing rule violations has not (cf Bennett-Moses and Chan 2015; Coglianese and Lehr 2016), and which I therefore focus upon. Pre-emptive algorithmic systems of this kind



14

(that is, algorithmic forms 3, 4, 7 and 8 outlined in Table 1) can be understood in light of various policy initiatives that have sought to influence the exercise of enforcement discretion strategically in the service of institutional goals, often supported by background ideological commitments in which synoptic oversight plays a key role (Ayres & Braithwaite 1998; Black 2005; Black 2006). So understood, pre-emptive algorithmic enforcement systems can be variously understood as a form of risk-based regulation, as a form of actuarial justice, and as a form of systematic surveillance-driven social sorting. a)

Pre-emptive enforcement as a form of risk-based regulation: So-called ‘risk-

based’ approaches to regulation have been championed by the UK government and other regulatory reform bodies over the last two decades. According to Beaussier and her colleagues, ‘the central conceit of risk-based regulation is that regulators cannot, and indeed should not even try, to prevent all possible harms’ (Beaussier et al 2016). Rather, regulatory interventions should focus on controlling the greatest potential threats to achieving regulatory objectives, as determined through ex ante assessments of their probability and consequences’ (ibid). Risk-based regulation has been most prominent at the level of inspection and monitoring in which risk-based approaches are adopted in order to identify potential violations and violators of regulatory standards. For example, the UK government now requires all regulators to adopt a ‘risk-based’ approach to the targeting of inspections in order to ease administrative burdens on regulated organisations and to ensure the proportionality of enforcement action (Rothstein & Downer 2012). Accordingly, various UK regulators have developed statistical surveillance tools which are intended to monitor the performance of regulated entities in order to identify those which are considered ‘high risk’ and thus prioritized for attention (Griffiths et al 2016). Algorithmic regulation of this kind can thus be understood as a form of ‘riskbased prioritization’ (per Griffiths et al 2016) which employs statistical prediction techniques to assist in the allocation of enforcement resources. b)

As a form of actuarial justice: A direct conceptual analogue to risk-based

regulation within the domain of criminological research is that of ‘actuarial justice’, which refers to a theoretical model within the criminal justice system that employs concepts and methods similar to actuarial mathematics. This approach to crime control departs from the conventional approach to the enforcement of the criminal law that prevailed in the UK and USA from the 1960s through to the early 1990s, which was overwhelmingly



15

reactive, dealing with criminal incidents as and when they are identified rather than strategic and pro-active in orientation (Ratcliffe 2016: 12-30). This new approach is evident in a range of policies and interventions used within many national criminal justice systems, including the use of actuarial methods to evaluate the risk and dangerousness of offenders and treatment programs, the adoption of crime prevention strategies and so-called ‘intelligence-led’ policing (Ratcliffe 2016). Closely associated with the work of Feely and Simon (1992, 1994), Robert identifies four characteristics of actuarial justice. First, crime is perceived as an ineliminable social fact so that the aim is not to eliminate crime but to reduce it to tolerable levels by assessing the risks that various situations and individuals pose. Crime has lost its moral component. Secondly, individual and social phenomena are reconstructed as risk objects, so that the focus of analysis is no longer the biographical individual, but one’s risk profile, created by reconstructing fragments of individual identity by combining variables associated with different categories and levels of risk. Thirdly, the aim is no longer that of transforming criminals into law-abiding citizens, but to manage the risks that offenders represent in which diagnosis and treatment give way to managerialism. Fourthly, its primary outlook and orientation is prospective, aimed at estimating and preventing the occurrence of criminal behavior rather than with sanctioning them or understanding and addressing their past causes (Robert 2005). Each of these characteristics resonates with the underlying logics of preemptive forms of algorithmic regulation, orientated towards intervention aimed at shaping the future by preventing unwanted activities, but arguably accentuated as the power of the risk profiling process is now greatly enhanced by the use of Big Data analytics. c)

As a form of surveillant driven social sorting: Both risk-based regulation and

actuarial approaches to criminal justice rest on a commitment to sorting and selecting individuals from a wider population in order to achieve more efficient and targeted enforcement action. The tasks of filtering and classification are fundamental logics of algorithmic regulation, enabling the automation of risk-based prioritization processes in which algorithms are applied to a regulated population to sort and classify individual entities within that population to enable the regulator to undertake more precise, targeted and efficient interventions. While computational risk classification processes of this kind have long been used in the context of insurance to assess whether to offer insurance coverage to individual applicants and at what price (Baker 2003), they have



16

also been employed on a more systematic, population-wide basis in the retail and marketing industry since the early days of data mining (Gandy 1993). These applications seek to enhance sales through more effective targeting of potential customers by utilizing data mining techniques to segment populations into different user groups and targeting them accordingly. It is through this process that objects of interest (whether individuals, organizations or – in the medical research context, for example – cancer mutations) can be ‘profiled’ and which have been attracted considerable attention from scholars of surveillance studies, and are examined more fully in section 5 below. 4.

Algorithmic Regulation as a Form of Social Ordering

Despite their differences, both reactive and pre-emptive forms of algorithmic regulation rest on a common underlying logic. Although O’Reilly is popularly credited with coining the term ‘algorithmic regulation’ in 2013 (O’Reilly 2013), it was already in scholarly use. In 2009, sociologist Aneesh Aneesh published findings of his ethnographic study of Indian workers providing IT and IT-enabled services to US firms, to investigate how the labour practice of ‘off-shoring’ is organised. Aneesh identified software programming schedules as critical to the organisation of globally dispersed labour through data servers, identifying a system of governance which he termed ‘algocracy’ based on ‘rule of the algorithm’ or ‘rule of the code’, that is distinct from both bureaucratic and market systems of social ordering. Although algocracy may have bureaucratic structures embedded in it (for example, specifying legally permissible operations for a teller while providing greater access to the same transaction available to the manager) Aneesh argued that the underlying software program is driven by the algorithm, or more deeply, the binary code, so that the imperatives of programming are not bureaucratic but mathematical (Aneesh 2009: 350). This characterisation of algorithmic systems as a distinct system of social ordering provides an enormously fruitful springboard for examination. Just as other forms of social ordering, such as markets and bureaucratic hierarchy, have been associated with a distinct set of political or ideological commitments, the following discussion considers how emerging algorithmic forms of social ordering might be associated with a particular set of political values and normative commitments. 4.1



The Politics and Drivers of Algorithmic Regulation: Dataism and its Discontents

17

Historian Yuval Harari refers to the underlying ideology of algorithmic regulation as ‘Dataism,’ rooted in a belief that humans can no longer distil the immense flows of data into information, knowledge or wisdom, so that the work of processing data should therefore be entrusted to computational algorithms, whose capacity far exceeds that of the human brain (Harari 2016: 368). While a critique of Harari’s Dataism is beyond the scope of this paper, his characterisation of Dataism as an ‘emerging religion’ that transcends differences in political ideology is worth noting. In particular, Harari claims that: free market capitalism and state controlled communism aren’t competing ideologies, ethical creeds or political institutions. At bottom, they are competing data-processing systems. Capitalism uses distributed processing, whereas communism relies on centralised processing. Capitalism processes data by directly connecting all produces and consumers to one another, and allowing them to exchange information freely and make decisions independently (Harari 2016: 369).

Dataism unites capitalism and communism within a single overarching vision by assuming that the ultimate goal of political systems is the optimal distribution of society’s material resources. But conceiving of political systems so crudely overlooks their underlying politics, values and premises. Algorithmic systems have themselves been associated with two dramatically opposed political visions. On the one hand, O’Reilly portrays algorithmic regulation as a seamlessly efficient, automated, data-driven approach that will enable us to resolve societal co-ordination problems with the technological prowess and efficiency of Google’s search engine (O’Reilly 2013), a view championed by the World Economic Forum which regards the personal digital data ecosystem as ‘vitally important to improving the state of the world’ (World Economic Forum 2011). But Evgeny Morozov’s stinging critique of Silicon Valley ‘solutionism’ offers an alternative political reading (Morozov 2014; Morozov 2013), seeking to expose what he claims is the hidden, anti-democratic political vision of Silicon Valley’s belief that technological innovation can effectively solve social problems simply by harnessing the power of the internet. For Morozov, Silicon Valley Solutionism bears four characteristics: First, deal with the effects of problems rather than their causes, preferably via internet-enabled apps, sensors and feedback loops – all provided by private sector start-ups – reflecting political assumptions in the logic of contemporary governing identified by Italian philosopher Giorgio Agamben (Morozov 2013; Agamben 2013). Secondly, expand oversight and



18

collect as much data as you can, thereby enhancing the power of regulators and increasing production of desirable outcomes, which is antithetical to the vision of the small libertarian state (O’Reilly 2013). Thirdly, encourage individuals to take responsibility for their own welfare, for example by maintaining their health via apps and digital platforms that monitor biometric indicators of health, and availing themselves of home sensing devices that automatically alert emergency services when threats are detected. Thus, more morally responsible individuals will utilise these technologies to improve their health, their personal security and their productivity, a logic that fits neatly with Foucault’s ‘governmentality’ through which the self-governing capabilities of individuals are brought into alignment with the political objectives of authorities, via the ‘responsibilisation’ of individuals, such that risks of illness, unemployment, poverty are regarded as matters for which the individual is responsible (Rose 2001; Lukes 2005: 91). Fourthly, individuals are best understood as entrepreneurs and the sharing economy as the new welfare state. Morozov’s critique springs from the observation that the means by which we seek to govern have inescapably political and ideological dimensions, lamenting the way algorithmic solutions are often presented ‘as if the very choice of how to achieve those “desired outcomes” was apolitical rather than a choice between different and often incompatible visions of communal living’ (Morozov 2014). For Morozov, Silicon Valley’s political vision characterises citizens primarily as entrepreneurs, small stockholders in a giant enterprise, empowered to take care of their own affairs thanks to ubiquitous digital feedback loops in which the role of the algorithmic state is to make reputation into a feedback-friendly social net so that honest hardworking individuals will generate high online reputations thereby producing a highly personalised social net that is ‘ultra stable’. Unlike the welfare state, the algorithmic state makes no assumptions about the existence of specific social problems requiring concerted collective action by the state, and if they exist, they can only be tackled by individual action, perhaps because, as Morozov observes, social injustice is much harder to track than the daily routines of those individuals it affects (Morozov 2014). 4.2

The Economics of Algorithmic Regulation: Surveillance Capitalism





19

When combined with the logic of capitalism, the ‘Solutionist’ mindset identified by Morozov has fostered the proliferation of algorithmic systems, driving a powerful logic of accumulation which Shoshana Zuboff dubs ‘surveillance capitalism’. Driven primarily by Silicon Valley’s hyperscale technology firms (and spearheaded by Google) which achieve growth mainly by leveraging automation via global digital platforms,10 the logic of surveillance capitalism rests on the generation of revenues from data assets acquired by ubiquitous automated operations, constituting a new asset class which Zuboff dubs ‘surveillance assets’ (Zuboff 2015: 80; WEF 2011). Investors (‘surveillance capitalists’) profit from the global networked environment through a new default business model where company valuations routinely depend upon ‘eyeballs’ rather than revenue as a predictor of return on investment, channelling and controlling flows of personal information while converting them to flows of profit, all in ways that are highly opaque to their users (Zuboff 2015). 5.

The New Surveillance and the Rise of Algorithmic Power

Surveillance capitalism relies critically on highly granular, population wide and continuously updated mass surveillance, all made possible by the networked digital infrastructure upon which industrialised societies increasingly rely. Continuous, real-time surveillance is critical to the operation of all forms of algorithmic regulation, whether reactive or predictive in orientation: see Table 1 above (Yeung 2017). But unlike the repressive forms of visual surveillance evoked by George Orwell’s Big Brother (Marx 2002), Julie Cohen argues that ‘[w]e are witnessing the emergence of a distinctly Western, democratic type of surveillance society’ (Cohen 2014) as we willingly allow ourselves to be subject to algorithmic surveillance by exchanging our personal data for the efficiency and convenience of networked digital tools (Harcourt 2014). This contemporary form of surveillance is undertaken primarily by the private sector, with Zuboff claiming that it is ‘Big Other’, rather than ‘Big Brother’ that we should fear (Zuboff 2015). At the same time, data protection scholars strenuously criticise the ‘notice and consent’ paradigm upon which the legality of ubiquitous mass algorithmic surveillance rests. They argue that, in the complex personal data eco-system which now exists, it is

10

Insights from scholars of surveillance studies also highlight how networked digital technologies have resulted in the growth of surveillance by individuals themselves, and is sometimes perceived as beneficial and even entertaining: Timan, Galic and Koops (2017).



20

practically impossible for individuals to provide meaningful, voluntary consent to the sharing and processing activities which algorithmic regulation entails (Acquisti, Braidmarte & Lowenstein 2015), particularly given that ‘ barter’ has emerged as the primary business model for the provision of contemporary digital services, with users agreeing to disclose their data in return for ostensibly ‘free’ services, (van Dijck 2014) thereby eliminating an important barrier to the adoption faced by firms seeking to attract new customers with initially high uncertainty about the valuation of the service offered (Lambrecht 2013). This underlying logic of exchanging privacy for convenience and efficiency also operates at the collective level, particularly as governments struggle with enormous pressure to pursue fiscal restraint and illustrated by the recent decision of three London NHS hospitals to allow Google DeepMind to access the NHS records of 1.6 million patients, in response to Google’s willingness to develop health analytics tools for NHS use, ostensibly on a pro bono basis. Yet our consent, both individually and collectively, may be more akin to that of the compulsive gambler who finds it impossible to kick the habit, unwilling to forgo the benefits which algorithmic regulation appears to offer despite the potentially harmful long term political, social, legal and moral costs (Yeung 2017). Meanwhile, the high priests of Silicon Valley dismiss concerns about privacy as anachronistic and irrelevant, evidenced by Sun Microsystems co-founder Scott McNealy’s proclamation, ‘You have no privacy anyway. Get over it’ (Sprenger 1999). 5.1

Legal critiques of algorithmic power

By willingly exchanging their personal data for the convenience of networked digital services, individuals may unwittingly increase their risk of exclusion from life critical opportunities. For example, Bill Davidow observes that both corporations and governments are employing big data analytics to create highly granular profiles upon which many consequential decisions are being made, such that millions of people are now virtually incarcerated in ‘algorithmic prisons,’ resulting in the inability of many people to find employment due to their internet profiles, or who have difficulty accessing a range of services including insurance, loan finance, rental housing, properties to purchase or to board an airplane (Davidow 2014). Even when individuals are aware of these automated processes and the tangible harm they might produce, Davidow points out that they may not know who their jailer is or even the type of behaviour that condemned them. These concerns are echoed by legal scholars who highlight how



21

algorithmic decision-making systems may antagonise constitutional and democratic values, including risks to informational privacy and other fundamental rights, and to procedural and substantive fairness, which are briefly outlined below. a) Informational privacy and fundamental rights: Among the most prominent legal concerns associated with the algorithmic turn focuses on the right to informational privacy, given the continuous monitoring and collection of personal digital data that many algorithmic systems necessitate. Several scholars have demonstrated how the technique of algorithmic profiling may undermine individual autonomy, selfdetermination and may risk subverting or antagonize civil and political rights such as the right to freedom from unfair discrimination (Ferraris et al 2014). Yet because profiling involves inferring a set of characteristics (typically behavioural) about an object of interest11 (Bygrave 2001), these profiles do not necessarily relate to any specific individual (Roosendaal 2010, 2013; Hildebrandt 2013). Rather, they are algorithmic constructions, abstractions of an individual made up of fragments of correlated variables identified from a match between some of her attributes with the profiles inferred from masses of attributes from masses of individuals, which Louise Amoore calls ‘data derivatives’ (Amoore 2013). In addition, the profiled individual is typically oblivious to how she is being profiled (or indeed that she is being profiled at all), and has no opportunity to understand the basis upon which assessments about her are made. Hence it is practically impossible for any individual to assert a claim of right, to protest that she is ‘not like that’, or that while she might have been like that in the past, she does not propose to be ‘like that’ in the future. In other words, concerns about the profiling process may not lend themselves easily to individual rights claiming, given the nature and structure of fundamental rights within contemporary western jurisprudence. While the consequences of inaccurate profiling may be trivial (such as unsuitable product recommendations generated by digital shopping recommendation engines) the same can hardly be said for individuals

11

The profiling process has two main components: first, the ‘knowledge discovery’ phase during which profiles are inferred by the analysis of large data sets to discover patterns, sequences, and relationships between data in order to arrive at a set of assumptions (the profile) based on probabilistic reasoning. This phase involves gathering data, pre-processing into a usable format, consolidating it and analysing it to discover what it may reveal in order to create a model of the variable of interest. The second phase entails the profile application phase, during which objects of interest/entities are treated in light of the profiles generated in the first phase of the process, in order to make decisions about that entity. In this phase, correlations discovered amongst data during the first phase are incorporated into an algorithm and applied to make predictions/decisions.



22

who are denied the freedom to board an airplane, to be released on bail, or denied access to employment or educational opportunities as a result of these highly opaque decisionmaking processes (Angwin et al 2016). b) Due Process: Other legal concerns highlight the lack of meaningful opportunities for affected individuals to contest algorithmic decisions. Constitutional rights of ‘due process’ (in the language of American constitutional law) or ‘procedural fairness’ (in British legal parlance) are intended to ensure that those affected by the decisions of public officials should be given an opportunity to participate in decisions that affect them, and that such decisions should be made in an unbiased manner (Craig 2012: Chapters 13-14). While the core principles of Anglo-American administrative law requiring executive discretion to be exercised lawfully, fairly and in accordance with the due process rights of individuals (Citron 2008) conventionally apply only to the exercise of governmental or public decision-making authority, there is rising anxiety about the freedom of powerful firms to act unilaterally against individuals without giving them an opportunity to contest or challenge such actions (Kerr 2010; Pasquale 2006; Pasquale and Bracha 2015; Searls 2012). Concerns about hidden bias in decision-making arise in relation to both human and automated forms of decision-making. One advantages of the latter is its capacity for consistency while avoiding the conscious and subconscious biases that invariably affects human decision-making. On the other hand, considerable attention has focused on the extent to which algorithmic decision-making systems may be affected by hidden, unintended biases which may be difficult to detect and which, owing to their capacity to operate on a one-to-many basis, like the Facebook News Feed algorithm, have the capacity to affect many millions of users at a single stroke. c) Substantive fairness, equality and distributive justice: The substantive quality and content of algorithmic decisions has also attracted alarm. Algorithmic decisions may be inaccurate or erroneous. Although algorithmic regulation holds out the prospect of applying objective, mathematically rigorous methods to solve social problems, by drawing on ‘raw data’ emitted directly from the environment itself, rather than reliant upon the subjective (and therefore fallible) observations and choices of humans, the notion of algorithmic objectivity and infallibility is a widely held myth (boyd and Crawford 2012). Data does not exist ‘out there’ in some kind of wild, natural state: it must be imagined as data in the first instance, a process which must have an interpretative base



23

(Gitelman 2011). Not only can errors arise in every stage of the process of algorithmic development, from the collection of data and their calibration, the filtering and interpretation of data, the choice of algorithmic technique, the design of the algorithm and the interpretation of algorithmic output, but these processes are inherently subjective (boyd and Crawford 2012). Secondly, algorithmic decision-making processes may render biased judgments, thereby perpetuating discrimination against historically disadvantaged groups (Executive Office of the President 2014, 2016).12 These concerns were identified over 20 years ago by Oscar Gandy who, building on Foucault’s work on panopticism, describes how the collection, processing and sharing of information about individuals and groups are used in the marketing and retail industries to coordinate and control access to goods and services (Gandy 1993). Gandy warned that, when used to profile individuals, these technologies are ‘discriminatory by design’, potentially undermining key aspects of democracy, equality, fairness and distributive justice, yet against which the law offers little protection (Gandy 1993; Danna and Gandy 2002). Even when considerable care is taken to avoid selection bias, algorithmic systems may unintentionally perpetuate and reinforce historical biases because the feedback loop that these systems rely upon draws on historic data patterns that may replicate themselves in algorithmic system outputs.13 5.2

Automation, human agency and the distribution of decision-making authority

In response, some might argue these concerns can be avoided if algorithmic systems are configured as recommender systems (that is, forms 2, 4, 6 and 8 identified in Table 1)

12

An extraordinary example of the discriminatory potential of algorithmic power is that of Facebook’s race specific ad content targeting in which individuals algorithmically identified as Black, Hispanic or White (the latter also including any non-Black and non-Hispanic individuals) were shown different movie trailers for the same film: http://thenextweb.com/insider/2016/03/18/facebook-showed-us-all-very-different-straight-outtacompton-trailers-based-on-race/. 13 For example, a recent study to simulate 1000 Google users (half male, half female) started with no search history and then had users visit the top 100 employment websites. Subsequently, on a third party news site, the study found that male users were shown high paying job advertisements 1800 times, while female users were shown high paying job advertisements only 300 times (Datta et al 2015). Similarly, stark racial disparities were found in a study of an algorithmic crime risk assessment tool developed by the private software firm Northpointe currently used by a number of US states within the criminal justice process. In one Florida county, blacks were almost twice as likely as whites to be labelled a higher risk but not actually re-offend, while whites were much more likely than blacks to be labelled lower risk but go on to commit other crimes (Angwin et al 2016).



24

rather than automated sanctioning systems (that is, forms 1, 3, 5 and 7 identified in Table 1) , because a human formally retains decision-making authority. For example, if preemptive algorithmic recommender systems offer guidance to law enforcement officials in prioritizing their limited inspection resources, officials can choose not to act upon them, and even if they do, the decision to investigate does not amount to a finding of guilt so that if a candidate algorithmically identified as high risk has not in fact violated the law, this should become apparent during subsequent investigation. In my view, however, conferring formal decision-making authority on a human does not overcome these concerns for at least four reasons. First, merely inserting a human into a decision-making procedure does not itself satisfy the twin requirements of due process, which are often described as the right to a fair hearing and the right to an unbiased tribunal (Galligan 1996). Secondly, even if law enforcement officials retain discretion to refrain from acting on algorithmic guidance, the power to investigate potential violations of the law cannot be exercised arbitrarily or otherwise unlawfully, even though the decision to investigate does not entail a determination of guilt, because merely singling out an individual for further investigation constitutes a form of unwanted adverse treatment (Ashworth 1998).14 In practice, those excluded from jobs, university places or other valuable opportunities by automated algorithmic systems are rarely informed of the underlying reasons for their exclusion, and typically unaware that the decision to exclude them was informed by algorithmic evaluation based on data correlations that may have no causal foundations whatsoever. But an individual’s lack of awareness that a decision affecting her is unfair does not render it acceptable. Thirdly, humans are highly susceptible to ‘automation bias’, referring to the tendency of individuals to defer to computational judgments, even when they are capable of recognizing that the situation calls for another choice (Carr 2015). Finally, algorithmic recommender systems constitute a very powerful form of choice architecture, shaping user perceptions and behavior in subtle but effective ways through the use of ‘hypernudge’ techniques, undermining an individual’s capacity to exercise independent discretion and judgment (Yeung 2017). In other words, although human agency might on occasion act to overcome or mitigate the

14

Indeed, criminological studies have highlighting how police stop and search powers have systematically discriminated against racial minorities and, as Bowling and Phillips observe, the way in which these powers have been exercised in the UK remain among the most contentious aspects of British policing (Bowling and Phillips 2007).



25

procedural and substantive concerns associated with the use of algorithmic decisionmaking systems, it cannot be systematically relied upon to do so. 5.3

Algorithmic Accountability and the Social Foundations of Democracy

The character of algorithms as opaque, inscrutable ‘black boxes’, owing to the sophisticated computational processes upon which they rely and their protection from disclosure as trade secrets (Pasquale 2015), yet capable of exercising or informing decision-making power with highly consequential effects, is fuelling demands to secure algorithmic accountability. As British constitutional scholar Dawn Oliver explains, ‘accountability is about requiring a person to explain and justify – against criteria of some kind – their decisions or acts, and then to make amends for any fault or error’ (Oliver 1994: 15). Meeting the requirement of explainability is particularly acute for decisionprocesses that rely on machine learning algorithms because they are based on patterns and correlations between data points, rather than on a causal or explanatory theory of behaviour, and are continuously reconfigured in light of past input and output data (Dourish 2016). Yet within liberal democratic societies, not only are individuals regarded as morally entitled to the reasons for decisions that adversely affect them, however imperfectly protected as a matter of law (Galligan 1996), but a liberal society aspires to be a transparent order in that its workings and principles should be well-known and available for public apprehension and scrutiny, resting on an enduring commitment (often expressed in the idea of a social contract) that the social order must be one that can be justified to those who live under it (Waldron 1987: 146). This is not to say, however, that values of transparency and accountability should necessarily trump the value of efficiency, but merely to argue that these value trade-offs should be openly debated rather than simply resolved in favour of efficiency by technological fiat. Concerns about the implications of the algorithmic turn for the collective values of transparency and accountability highlight how a wholesale shift towards algorithmic decision-making systems risks eroding the collective moral and cultural fabric upon which democracy and individual freedom rests, thereby undermining the social foundations of liberal democratic political orders (Yeung 2011). Some of the most prescient critiques of the rise of algorithmic power point out that an unduly narrow focus on privacy (understood as an individual right to control the information about one’s self) fails to



26

recognise the extent to which many civil and political rights are ultimately rooted in a set social and political structures in which privacy is also understood as a collective good. Hence the contemporary legal model of privacy self-management is inadequate because it effectively characterises informational privacy as a marketised commodity that can be traded off at will to the highest bidder, reflected in the critical legal role played by on-line notice and consent procedures. But privacy refers to a zone of protection around each individual’s activities within a society that makes possible the capacity for individual flourishing and self-creation, that allows us to play around with who we are, with whom we wish to relate and on what terms, and in which our sense of our self and our individuality can emerge, mutate and stabilise (Cohen 2012). If a sufficiently large critical mass of individuals decide, independently, that they are willing to waive their rights to privacy in return for convenience and efficiency, then the resulting social structure cannot sustain a ‘privacy commons’ and cannot therefore provide a zone of privacy for its members, including those who choose not to waive their individual rights to privacy. Yet the importance of this critical moral and social infrastructure that is vital to human flourishing and democratic freedom is frequently overlooked in contemporary debates. In contrast, legal philosopher Mireille Hildebrandt argues that within constitutional democracies, sovereign rule is rendered legitimate through a double form of transparency: first, people live under rules of their own making (ie democratic participation), and secondly, the application of those rules can be contested in a contradictory procedure that is capable of opening the black box of their interpretation (the rule of law). It is these two elements through which modern legal systems establish one of the most successful (albeit imperfect) cybernetic systems: constitutional democracy. It is a system of governance that rests on a series of checks and balances which institutes a perfect feedback loop that operates in two directions between the rulers and the ruled. Hence all who live under the rule of law are not regarded as mere objects to be controlled, but subjects participating in collective self-rule, accountable for their actions, to their government, and to each other (Hildebrandt 2016). The turn to algorithmic regulation and the data-driven agency which it sustains threatens this equilibrium. By continuously tracking of individuals at a highly granular level, algorithmic regulation operates as a one-way mirror that allows those looking down to surveil those below, but who lack any realistic prospect of peering into, let alone comprehending, the algorithmic black boxes that increasingly regulate our daily lives. Similarly, Zuboff’s



27

assessment of a computer-mediated world painted by Google’s Hal Varian could scarcely be more chilling, claiming: …Varian’s vision of a computer-mediated world strikes me as an arid wasteland – not a community of equals bound through laws in the inevitable and ultimately fruitful human struggle with uncertainty. In this futurescape, the human community has already failed. It is a place adapted to the normalization of chaos and terror where the last vestiges of trust have long since withered and died. Human replenishment for the failures and triumphs of asserting predictability and exercising our will in the face of natural uncertainty gives way to the blankness of perpetual compliance... It is a ubiquitous networked institutional regime that records, modifies, and commodifies everyday experience from toasters to bodies, communication to thought, all with a view to establishing new pathways to monetization and profit. Big Other is the sovereign power of a near future that annihilates the freedom achieved by the rule of law (Zuboff 2015: 8182).

Common to both critiques is a concern about the asymmetrical nature of the power relationships that are implicit in the way in which algorithmic regulation typically operates owing to powerful commercial incentives that drive the design and operation of digital platforms. In the examples of computer-mediated contracts contemplated by Varian, automatic enforcement appears to operate in favour of only one party to the contract (the car rental firm but not the individual hirer; the lending company but not the individual borrower) constituting what Friedrich Kessler called ‘contracts of adhesion’, offered by powerful service providers to individuals who wish to access the service on a ‘take it or leave it’ basis (Kessler 1943) and ‘nailing down the submissive party while the dominant party is free to charge whatever it wants’ (Searls 2012: 54). Nevertheless, it seems theoretically possible that systems of algorithmic regulation might be imagined and designed in ways that are more egalitarian and progressive in their values, orientation and operation. In particular, computer-mediated contracts could allow opportunities for bilateral participation in the setting of the terms of the agreement through meaningful negotiation and bargaining between the parties, so that it is only after consensual terms are agreed that their performance is monitored and automatically enforced. In this way, computer-mediated contracts could operate on a more reciprocal basis, in which automated enforcement of contractual terms operate for the benefit of both contracting parties. Thus, we could modify Varian’s examples, reimagining a world in which the car rental company monitors the customer’s care and management of the vehicle whilst, at the same time, the customer relies on computer mediated feedback to monitor the supplier’s care of the fleet, how often breakdowns



28

and defects are logged, how quickly it takes the supplier to resolve and repair them, and perhaps even the prices and terms offered to other customers. This would allow for more agile, flexible contracts. Thus the holiday-maker based in a specified location on a two-week holiday could hire a vehicle through a series of one-day contracts. At the end of each day, if the customer’s algorithmically-assisted assessment of the car hire company’s stewardship of the fleet, and the terms and price offered for the following day are sufficiently attractive, the contract would then be automatically renewed for an additional day – if not, then the contract could be automatically switched to an alternative supplier.15 In other words – a computer-mediated world might, in other words, seek to establish two-way mirrors, rather than the one-way mirrors that are currently proliferating. Yet computer-mediated contracts of this reciprocal, bilateral kind do not appear to be the kind which Silicon Valley start-ups are competing with each other to construct. This is hardly surprising, for they do not offer lucrative financial returns of the kind that accrue to giant digital platforms that wield asymmetrical power vis-à-vis their users, making it difficult to imagine how the future digital economy might develop organically to resemble the much more genuinely consumer-empowered vision offered by Doc Searls’ so-called ‘intention economy’ (Searls 2012). 6.

Conclusion

Whether one shares Zuboff’s pessimism or O’Reilly’s optimism, the emergence of algorithmic regulation evokes rich and increasingly urgent questions which could be pursued through many disciplinary perspectives and analytical lenses. A recurring and important theme in scholarly and policy discussions focuses on the need to secure ‘algorithmic accountability’. But to understand what securing meaningful algorithmic accountability might require, we need to acquire a deeper understanding of algorithmic power. This paper has sought to analyse algorithmic regulation from the perspective of regulatory governance studies and legal and social scientific insight more generally. In so doing, I have done two things: first, I have undertaken a descriptive analysis of the phenomenon of algorithmic regulation, offering a definition of algorithmic regulation, classifying these systems as either reactive or pre-emptive, and offering a taxonomy that

15

Of course there would be switching costs associated with returning the vehicle to the car rental company and taking delivery of the new one that the customer would take into account in deciding whether to switch.



29

identifies eight forms of algorithmic regulation, based on how these systems are configured in relation to each of the three components of a cybernetic systems: that is, at the level of standard-setting (whether behavioural standards are ‘simple’/fixed or ‘complex’/adaptive), information gathering and monitoring (reactive or pre-emptive) and behaviour modification (automated or recommender systems). I have argued that while simple forms of algorithmic regulation display continuity with long-established approaches to control which seek to harness architecture and design to shape social behaviour and outcomes, the drastic reduction in cost associated with utilising automated digital systems to regulate social behaviour means that it has now become practically feasible for regulators, public administrators and others to adopt these systems in order to achieve their policy goals. On the other hand, complex forms of algorithmic regulation that rely on machine learning algorithms powered by huge data sets are genuinely novel, especially when configured as predictive systems that seek to pre-empt future behaviour. These control systems are vastly more powerful than conventional forms of architectural regulation (such as the speed hump or the door lock, for example) because they can track and intervene in the behaviour of not just a single user, but an entire population of users across a widely dispersed geographic area, whilst collecting and analysing real time population-wide data to predict, and act pre-emptively to forestall, both individual and population-wide decisions and actions. Secondly, I have mapped some of the critical debates surrounding algorithmic regulation, drawing on regulatory governance studies, legal scholarship and social scientific reflection more generally. In so doing, I have identified various dimensions of algorithmic regulation that have provoked concerns about its legitimacy, drawing attention to the rise of algorithmic power, the competing ideological visions of how algorithmic regulation should be understood, the logics and assumptions underpinning their expected operation, and growing concerns about the importance of devising mechanisms for ensuring algorithmic accountability given their capacity to expand, reinforce and redistribute power, authority and resources, and their capacity to affect the critical life opportunities of individuals. My examination has highlighted some of the deeper legal, social, democratic and moral implications for both individuals and societies of the algorithmic turn, demonstrating that there is much more at stake than informational privacy, including core legal and constitutional principles that are ultimately rooted in the liberal commitment to treat individuals with fairness and dignity and which,



30

in liberal democratic societies, include principles of transparency and accountability, due process and the rule of law (Brownsword 2016). In short, I have attempted to identify what is at stake in the rise of algorithmic power, highlighting various claims by those who emphasise its potential benefits, and those who warn of its dangers, while refraining from offering my own position on them. In so doing, my aim has been to raise more questions than answers in order to foster further research, dialogue and debate in the hope that others will continue to investigate algorithmic regulation as an important and, as I have hopefully demonstrated, potentially troubling, rapidly emerging and increasingly powerful governance phenomenon.



31

References: Agamben, G. (2013) ‘For a Theory of Destituent Power’, Public Lecture, Athens. 16 November. Available at http://www.chronosmag.eu/index.php/g-agamben-for-a-theoryof-destituent-power.html. Aneesh, A. (2009) Global Labor: Algocratic Modes of Organisation. Sociological Theory 27: 347-70. Acquisti, A., Brandimarte, L., Lowenstein, G. (2015) Privacy and Human Behavior in the Age of Information. Science 347( 6221): 509-14. Amoore, L. (2013) The Politics of Possibility. Duke University Press: North Carolina. Angwin, J., Larson, J., Mattu, S., and Kirchne, L. (2016) Machine bias. Pro Publica available at https://www.propublica.org/article/machine-bias-risk-assessments-in-criminalsentencing Ariely, D. (2009) Predictably Irrational: The Hidden Forces That Shape Our Decisions. London: Harper Collins. Arnoldi, J. (2016) ‘Computer Algorithms, Market Manipulation and the Institutionalization of High Frequency Trading’ Theory, Culture & Society 33(1): 29-52. Ashworth, A. (1998) The Criminal Process: An Evaluative Study. 2nd ed. Oxford: Oxford University Press. Ayres, I., and Braithwaite, J. (1992) Responsive Regulation: Transcending the Deregulation Debate. New York: Oxford University Press. Baldwin, R., Cave, M. and Lodge, M. (eds.) 2010. The Oxford Handbook of Regulation. New York: Oxford University Press. Baker, T (2003) Containing the Promise of Insurance: Risk Classification and Adverse Selection. In Ericson, R. V. and Doyle, A. (eds.) Risk and Morality 258-83. University of Toronto Press: Toronto. Beer, D. (2017) ‘The Social Power of Algorithms’. Information, Communication & Society. 20(1): 1-13. Bennett Moses, L., Chan, J. (2015) Algorithmic Prediction in Policing: Assumptions, Evaluation and Accountability. Paper presented at Associations in Global Law and Policy Workshop. Sydney, 10-12 December. Beniger, J. (1986). The control revolution: Technological and economic origins of the information society. Cambridge, Mass.: Harvard University Press. Beaussier, A-L, Demeritt, D, Griffiths, A & Rothstein, H (2016) Accounting for Failure: RiskBased Regulation and the Problems of Ensuring Healthcare Quality in the NHS. Health, Risk and Society 18: forthcoming.



32

Bevan, G., and Hood, C. (2006) What’s Measured Is What Matters: Targets and Gaming in the English Public Health Care System. Public Administration 84 (3): 517. Black, J. (2005) ‘The Emergence of Risk-Based Regulation and the New Public Risk Management in the United Kingdom’ Public Law. 512-48. Black, J. (2006) ‘Managing Regulatory Risks and Defining the Parameters of Blame: A Focus on the Australian Prudential Regulation Authority’ Law and Policy 28(1): 1-26. Black, J. (2008) Constructing and Contesting Legitimacy and Accountability in Polycentric Regulatory Regimes. Regulation & Governance 2(2): 137–64. Black, J. (2014) Learning from Regulatory Disasters. LSE Law, Society & Economy Working Papers 24/2014 available at http://eprints.lse.ac.uk/60569/1/WPS2014-24_Black.pdf. boyd, d., Crawford, K. (2012) Critical Questions for Big Data. Information, Communication and Society. 15 (5): 662-7. Braithwaite, J., Coglianese, C. & Levi-Faur, D. (2007) Can Regulation and Governance Make a Difference? Regulation & Governance 1 (1): 1-7. Brownsword, R. (2005) ‘Code, Control, and Choice: Why East Is East and West Is West’. Legal Studies 25:1-20. Brownsword, R. (2016). ‘Technological Management and the Rule of Law’ Law, Innovation and Technology 8(1): 100-40. Burrell, J. (2016) How the Machine ‘Thinks’: Understanding Opacity in Machine Learning Algorithms. Big Data & Society 3 (1) doi 10.1177/2053951715622512. Bygrave, L. (2001) Minding the Machine: Article 15 of the EC Data Protection Directive and Automated Profiling. Computer Law & Security Report 17: 17-21. Carr, N. (2015) The Glass Cage: Where Automation Is Taking Us. Random House: London. Citron, D. K. (2008) Technological Due Process. Washington University Law Review 85: 1249-31. Coglianese, C., Lehr, D,. (2016) Regulating by Robot: Administrative Decision-Making in the Machine-Learning Era. Georgetown Law Journal, forthcoming. Cohen, J. E. (2012) Configuring the Networked Self. Yale University Press: New Haven. Cohen, J. E. (2014) Between Truth and Power. In Hildebrandt, M. and van den Berg, B. (eds) Freedom and Property of Information: The Philosophy of Law Meets the Philosophy of Technology. Routledge: Abingdon. Craig, P. P. (2012) Administrative Law. 7th ed. Sweet & Maxwell: London. Cheney-Lippold, J. ‘A New Algorithmic Identity: Soft Biopolitics and the Modulation of Control’. Theory, Culture & Society 28, no. 5 (2011): 164-81.



33

Danna, A, Gandy Jr, O. H. (2002) ‘All That Glitters Is Not Gold: Digging beneath the Surface of Data Mining’. Journal of Business Ethics 30: 373-86. Datta, A., Tschantz, M.C., & Datta, A. (2015) ‘Automated Experiments on Ad Privacy Settings’. Proceedings on Privacy Enhancing Technologies 1: 92-112. Davidow, B. (2014) ‘Welcome to Algorithmic Prison - the Use of Big Data to Profile Citizens Is Subtly, Silently Constraining Freedom.’ The Atlantic, 20 February 2014 Demeritt, D., Rothstein, H., Beaussier, A-L & Howar, M. (2015) ‘Mobilizing Risk: Explaining Policy Transfer in Food and Occupational Safety Regulation in the UK’ Environment and Planning 47: 373-91. Dourish, P. (2016) ‘Algorithms and Their Others: Algorithmic Culture in Context’ Big Data & Society 3: 2 doi.org/10.1177/2053951716665128 Edwards, M.R., and Edwards, K. (2016) Predictive HR Analytics: Mastering the HR Metric. Kogan Page: London. Ekbia, H., Mattioli, M, Kouper, I., Arave, G., Ghazinejad, A, Suri, R., Tsou, A., Weingart, S. and Sugimot, C. R. (2015) ‘Big Data, Bigger Dilemmas: A Critical Review’ Advances in Information Science 68(8): 1523-45. Executive Office of the President (2014) Big Data: Seizing Opportunities, Preserving Values. Washington. The White House, Washington. Executive Office of the President (2016) Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights. The White House: Washington. Ferraris, V., Bosco, F. & D’Angelo, E. (2013) The Impact of Profiling on Fundamental Rights, Working Paper 3, available via SSRN Network. Feely, M., & Simon, J (1992) ‘The New Penology: Notes on the Emerging Strategy of Corrections and Its Implications’. Criminology 30: 449-74. Feely, M., and Simon, J. (1994) ‘Actuarial Justice’. In Nelken, D. (ed) The Futures of Criminology 173-201. Sage: London. Frizell, S. ‘A Historical Argument against Uber: Taxi Regulations Are There for a Reason’ Time Magazine, November 19 2014. Fuller, L. L. (1964) The Morality of Law. Yale University Press: New Haven. Galligan, D.D. (1996) Due Process and Fair Procedures. Oxford University Press: Oxford. Gandy, O.G. (1993) The Panoptic Sort. Westview: Boulder. Geradin, D. (2015) ‘Should Uber Be Allowed to Compete in Europe? And If So How?’, SSRN Network. Available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2615530



34

Gillespie, T. (2013) ‘The Relevance of Algorithms.’ In Gillespie, T, Boczkowski, P & Foot, K. (eds.) Media Technologies: Essays on Communication, Materiality and Society, Cambridge. MIT Press: MA. Gillespie, T. (2014) Algorithms (draft) (♯digital keywords). available at http://culturedigitally.org/2014/06/algorithm-draft-digitalkeyword/ Gitelman, L. (2013) Raw Data Is an Oxymoron. MIT Press: Boston. Griffiths A., Beaussier A-L., Demeritt D., & Rothstein, H. (2016) ‘Intelligent Monitoring? Assessing the Ability of the Care Quality Commission’s Statistical Surveillance Tool to Predict Quality and Prioritise NHS Hospital Inspections’. BMJ Qual Saf doi:10.1136/bmjqs2015. Harari, Y. (2016) Homo Deus - A Brief History of Tomorrow. London: Penguin. Harcourt, B. E. (2014) ‘Governing, Exchanging, Securing: Big Data and the Production of Digital Knowledge’. Columbia Public Law Research Paper No 14 available via SSRN network. Hildebrandt, M. (2016) ‘Law as Information in the Era of Data-Driven Agency’. Modern Law Review 79(1): 1-30. Hood, C., and Margetts, H. (2007) The Tools of Government in the Digital Age. Basingstoke: Palgrave Macmillan. Hood, C., Rothstein, H and Baldwin, R (2001). The Government of Risk. Oxford: Oxford University Press. Jurgenson, N. (2014) ‘View from Nowhere - on the Cultural Ideology of Big Data’. The New Inquiry, 9 October. Kerr, I. (2010) ‘Digital Locks’ and the Automation of Virtue. In Geist, M (ed) ‘Radical Extremism’ to ‘Balanced Copyright’: Canadian Copyright and the Digital Agenda. Irwin Law Toronto. Kessler, F. (1943) ‘Contracts of Adhesion - Some Thoughts About Freedom of Contract’. Columbia Law Review 54(5): 631. Kitchin, R. (2014a). ‘Big Data, New Epistemologies and Paradigm Shifts’ Big Data & Society 1-12. Kitchen, R. (2014b). The Data Revolution. London: Sage. Kitchen, R., Lauriault, T.P. and McArdle, G. (2015) ‘Urban Indicators and Dashboards: Epistemology, Contradictions and Power/Knowledge.’ Regional Studies, Regional Science 2 (1): 43-45. Krasnow Waterman, K., Bruening, P.J. (2014) ‘Big Data Analytics: Risks and Responsibilities’. International Data Privacy Law 4: 89-95.



35

Kuchler, H. (2014) ‘Data Pioneers Watching Us Work’. The Financial Times, February 17. Kysar, D.A. (2004) ‘Preferences for Processes: The Process/Product Distinction and the Regulation of Consumer Choice’. Harvard Law Review 118: 525. Lambrecht, A. (2013) ‘The Economics of Pricing Services Online’. In Surlauf, S. N. & Blume, L.E. (eds.) The New Palgrave Dictionary of Economics. Lessig, L. (1999) Code and Other Laws of Cyberspace. Basic Books: New York. Lukerson, V. (2015). ‘Here’s How Facebook’s News Feed Actually Works’ Time Magazine, July 9. Lukes, S. (2005) Power – A Radical View (2nd ed) Palgrave Macmillan: Hampshire. Lv, Y., Duan, Y.D., Kang, W., Li, Z., and Wang, F. (2015) ‘Traffic Flow Prediction with Big Data: A Deep Learning Approach’ IEEE Transactions on Intelligent Transportation Systems 16(2): 865-73. Lyon, D. (2007) Surveillance Studies. Cambridge: Polity Press. Mackenzie, A. (2015) ‘The Production of Prediction: What Does Machine Learning Want?’ European Journal of Cultural Studies 18: 429-45. Marx, G.T. (2002) ‘ What's New About the 'New Surveillance'? Classifying Change and Continuity’. Surveillance & Society 1(1): 9-29. Mayer-Schonberger, V. and Cukier, K. (2013) Big Data. London: John Murray. Medina, E. (2015) ‘Rethinking Algorithmic Regulation’ Kybernetes 44 (6/7): 1005-19. Miller, A. A. (2014) ‘What Do We Worry About When We Worry About Price Discrimination? The Law and Ethics of Using Personal Information for Pricing.’Journal of Technology Law & Policy 41-95. Morgan, B. & Yeung, K. (2007) An Introduction to Law and Regulation. Cambridge University Press: Cambridge. Morozov, E. (2013) To Save Everything, Click Here. Penguin Group: London. Morozov, E. (2014) ‘The Rise of Data and the Death of Politics’. The Guardian, 20 July. Murray, A., and Scott, C. (2002) ‘Controlling the New Media: Hybrid Responses to New Forms of Power’. Modern Law Review 65:491. NESTA (2015). Machines That Learn in the Wild. NESTA: London. Available at http://www.nesta.org.uk/sites/default/files/machines_that_learn_in_the_wild.pdf Ogus, A (1994) Regulation - Legal Form and Economic Theory. Clarendon Law Series. Oxford University Press: Oxford.



36

Oliver, D (1994) ‘Law, Politics and Public Accountability. The Search for a New Equilibrium.’ Public Law 238. O'Neil, C (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Allen Lane: New York. O'Reilly, T. (2013) Open Data and Algorithmic Regulation’ In Goldstein, B & Dyson L (eds) Beyond Transparency - Open Data and the Future of Civic Innovation, 289-300. Code for America Press: San Francisco. O'Rourke, D. (2003) ‘Outsourcing Regulation: Analyzing Nongovernmental Systems of Labor Standards and Monitoring’. Policy Studies Journal. 31: 1. Owen, T. (2015) ‘The Violence of Algorithms’ Foreign Affairs, 25 May. Pasquale, F. (2006) ‘Rankings, Reductionism, and Responsibility’. Cleveland State Law Review 54:115-38. Pasquale, F. (2015) The Black Box Society. Harvard University Press: Boston. Pasquale, F., & Bracha, O. (2015) ‘Federal Search Commission? Access, Fairness and Accountability in the Law of Search’. Cornell Law Review 93: 1149-91. Perrin, B. (2015) ‘Bringing Accountability up to Date with the Realities of Public Sector Management in the 21st Century’. Canadian Public Administration 58 (1): 183-2. Ratcliffe, J.H. (2016) Intelligence-Led Policing. Abingdon, UK: Routledge, 2016. Raz, J. (1977) The Rule of Law and Its Virtue. Law Quarterly Review 93: 195. Rieder, G., and Simon, J. (2016) ‘Datatrust: Or, the Political Quest for Numerical Evidence and the Epistemologies of Big Data’ Big Data & Society 3:1 doi.org/10.1177/2053951716649398 Robert, D. (2005) ‘Actuarial Justice’. In Bosworth, M (ed.) Encyclopedia of Prisons & Correctional Facilities. Sage: London. Roosendaal, A. (2014) Digital Personae and Profiles as Representations of Individuals. In Bezzi, M., Duquenoy, P., Fishcher-Humbner, S., Hansen, M. & Zhang, G. (eds) Privacy and Identity Management for Life. Springer: Dordecht. Rose, N. (2001) ‘The Politics of Life Itself’. Theory, Culture & Society 18 (6): 1. Rothstein, H., & Downer, J. (2012) ‘Renewing Defra’: Exploring the Emergence of RiskBased Policymaking in UK Central Government. Public Administration 90: 781-99. Searls, D. (2012) The Intention Economy: When Customers Take Charge. Harvard Business Review Press: Boston. Silver, D. (2016) Deep Reinforcement Learning. Available at https://deepmind.com/blog.



37

Social Media Collective Research Blog, ‘Critical Algorithm Studies: A Reading List’ available at https://socialmediacollective.org/reading-lists/critical-algorithm-studies/ (accessed 11 November 2016) Solove, D. J. (2013) ‘Privacy Self Management and the Consent Dilemma’. Harvard Law Review 126: 1880-93. Sprenger, P. (1999) Sun on Privacy: 'Get over It'. Wired, 26 January. Timan, T., Galic, M. and Koops, B-J. (2017) ‘Suveillance: Theory, Law and Regulatory Implications’ in Brownsword, R., Scotford, E., and Yeung, K. (eds), The Oxford Handbook of Law and the Regulation of Technology, Oxford University Press: Oxford, forthcoming van Dijck, J (2014) ‘Datafication, Dataism and Dataveillance’: Big Data between Scientific Paradigm and Ideology. Surveillance & Society 12(2): 199-208. Varian, HR (2014) ‘Beyond Big Data’. Business Economics 49(1): 31. Waldron, J (1987) ‘Theoretical Foundations of Liberalism’. The Philosophical Quarterly 37: 127-50. Williamson, O.E. (1975) Markets and Hierarchies: Analysis and Antitrust Implications : A Study in the Economics of Internal Organization. Collier Macmillan: London. Wood, A. (2016) ‘Digital Transformations of Work: What Is Driving the Digital Transformation of Work? Union Solidarity International’. Available at https://usilive.org/opinions/digital-transformations-of-work-technology-power-culturewhats-driving-the-digital-transformation-of-work/ World Economic Forum (2011) Personal Data – The Emergence of a New Asset Class, WEF, Geneva. World Economic Forum (2014) Rethinking Personal Data, WEF, Geneva. Yeung, K. (2008) ‘Towards an Understanding of Regulation by Design’. In R Brownsword and K Yeung (eds.) Regulating Technologies, 79-94. Portland, Oregon: Hart Publishing. Yeung, K., & Dixon-Woods, M. (2010) ‘Design-Based Regulation and Patient Safety: A Regulatory Studies Perspective’. Social Science and Medicine 71: 502-09. Yeung, K. (2011) ‘Can We Employ Design-Based Regulation While Avoiding Brave New World’ Law, Innovation and Technology 3(1): 1-29. Yeung, K. (2015). ‘Design for Regulation’ In van den Hoven, J., van de Poel, I. and Vermaas, P.E. (eds.) Handbook of Ethics, Values and Technological Design. Dordecht: Springer, 2015. Yeung, K. (2017) ‘Hypernudge’: Big Data as a Mode of Regulation by Design’. Information, Communication & Society 20(1) 118-136. Zittrain, J. (2009) The Future of the Internet. Penguin Books: London.



38

Ziewitz, M. (2016) ‘Governing Algorithms: Myth, Mess, and Methods’ Science, Technology & Human Values 41 (1): 3-16. Zuboff, S. (2015) Big Other: Surveillance Capitalism and the Prospects of an Informal Civilization. Journal of Information Technology 30: 75-89. (Revised 13072 words- everything) (2267 words in references) = 10,805 words in main text

View publication stats

39