Big Data for Managers: Creating Value 1138593087, 9781138593084

Table of contents :
Cover
Half Title
Title Page
Copyright Page
Dedication
Table of Contents
Acknowledgements
Foreword
1. Introduction
For the practising manager
A non-technical book
Structure of the book
2. Big Data revolution
Data driven decisions and value creation
History of data and Big Data
Data and analysis
Data analysis and statistics
Data analysis and computing
Advances in collection mechanisms
Relational databases
Data warehouse and business intelligence
Data mining
Google web search
Big Data analysis on the cloud
Structured data
Unstructured data
Big Data
Customer data
Employee data
Competition data
Summary
Notes
3. Creating value from Big Data
Value drivers in commercial organizations
The proforma income statement and balance sheet
Market value and non-financial values
Investments where Big Data can create value
Creating value from additional revenue from existing customers – higher prices
Creating value from customer satisfaction
Creating value from reduced subscriber acquisition costs
Creating value from more effective marketing – acquiring more customers
Creating value from customer retention value
When data itself is the source of value creation
Summary
Notes
4. Big Data techniques and solutions
Big Data analytics
Operational excellence
Smart decision making
Innovation
Data analytics techniques
Statistical analysis
Social network analysis
Semantics
Data visualization
Predictive analysis
Summary
Notes
5. Introducing the model: design and implementation
C-ADAPT model of Big Data value creation
1. C (Challenges and/or goals)
2. A (Areas identification)
3. D (Data discovery)
4. A (Analysis and insights)
5. P (Presentation and visualization)
6.T (Testing the value created, refine and repeat)
C-ADAPT worksheet
C – Challenges and goals
A – Areas related
D – Data discovery
A – Analysis and insights
P – Presentation and visualization
T – Testing the value created, refine and repeat
Summary
Notes
6. Big Data case studies
Ooredoo (formerly Qtel)
Rebranding and message amplification
Domino’s Pizza
Understanding buying behavior
Leading antivirus company
Increasing engagement and driving revenue
Gate Gourmet
Use of data for competitive analysis
Tesco
Knowing customer insights from data
Delta Airlines
Data is helping them find lost baggage
Intel
Saving manufacturing costs with Big Data
TXU Energy
Saving costs by using smart electric meters
OmedaRx
Big Data to improve medication adherence
John Deere
Revolutionizing farming using Big Data
Airbnb
Price recommendations using Big Data
Walmart
Smart searching using semantic analysis
Huffington Post
Using Big Data to drive the traffic
Summary
Notes
7. What practitioners say
Big Data is important – very important!
Key value from Big Data
Challenges in implementing Big Data projects
Summary
Notes
8. Conclusion and discussion
Index

Citation preview

BIG DATA FOR MANAGERS

In today’s fast growing digital world, the web, mobile, social networks and other digital platforms are producing enormous amounts of data that hold intelligence and valuable information. Correctly used it has the power to create sustainable value in different forms for businesses.The commonly used term for this data is Big Data, which includes structured, unstructured and hybrid structured data. However, Big Data is of limited value unless insightful information can be extracted from the sources of data. The solution is Big Data analytics, and how managers and executives can capture value from this vast resource of information and insights. This book develops a simple framework and a non-technical approach to help the reader understand, digest and analyze data, and produce meaningful analytics to make informed decisions. It will support value creation within businesses, from customer care to product innovation, from sales and marketing to operational performance. The authors provide multiple case studies on global industries and business units, chapter summaries and discussion questions for the reader to consider and explore. Big Data for Managers also presents small cases and challenges for the reader to work on – making this a thorough and practical guide for students and managers. Atal Malviya is CEO at Spark10 and Adjunct Faculty at Ashridge Executive Education at Hult International Business School, UK. He serves on the boards of many public and private companies. A passionate innovator, he writes and speaks about Entrepreneurship, Start-up Investing, Creating Value, Big Data, Blockchain, Machine Learning and Artificial Intelligence. Atal has an MBA from Ashridge Business School and MS Computer Science from India. Dr Mike Malmgren is Associate Professor at Ashridge Executive Education at Hult International Business School, UK and at Linköping University, Sweden. His interest is strategy innovation and how companies can create value. In addition, Mike has extensive experience as Managing Director for several technology companies in Sweden, the US, Australia and the UK.

“Digitization affects all companies. One of the biggest challenges for business executives is to apply the technical capabilities of Big Data into daily business. With its management perspective, its structured model and several interesting case studies, this book provides very good support for managers in how to use Big Data in their business development.” Mats Abrahamsson, Professor, Linköping University, Sweden “Big Data for Managers does a great job of introducing analytics concepts for an audience that rapidly needs to understand how Big Data can both transform their business but, equally, make it irrelevant, fast.” Christopher Ahlberg, CEO Recorded Future and Chairman, Hult International Business School “A well written book, with a good structure. The contents are well laid out, making it very easy for someone to refer to a certain point. The case studies are valuable – puts Big Data knowledge into application. There is a range of companies, and each case study focuses on a certain efficiency Big Data usage has brought about, with a clear structure of background, opportunities, methods and results.” Manoj Gupta, Vice President, HCL Technologies, UK “Proven use cases from different industries will help business managers, charting clear path towards data driven work culture in their organizations.” Mahendra K. Upadhyay, Head of Data & Technology, Mindshare India “In a data-driven era, this book is an excellent overview everything that you must know about Big Data analytics.” Amit Chandak, CTO, Progen Business Solution (BI), India

BIG DATA FOR MANAGERS Creating Value

Atal Malviya and Mike Malmgren

First published 2019 by Routledge 2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN and by Routledge 52 Vanderbilt Avenue, New York, NY 10017 Routledge is an imprint of the Taylor & Francis Group, an informa business © 2019 Atal Malviya and Mike Malmgren The right of Atal Malviya and Mike Malmgren to be identified as authors of this work has been asserted by them in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library Library of Congress Cataloging-in-Publication Data Names: Malviya, Atal, 1978- author. | Malmgren, Mike, 1958- author. Title: Big data for managers : creating value / Atal Malviya and Mike  Malmgren. Description: 1 Edition. | New York : Routledge, 2019. | Includes   bibliographical references and index. Identifiers: LCCN 2018022753 (print) | LCCN 2018038164 (ebook) |   ISBN 9780429489679 (eBook) | ISBN 9781138593060 (hardback:   alk. paper) | ISBN 9781138593084 (pbk. : alk. paper) | ISBN   9780429489679 (ebk) Subjects: LCSH: Big data. | Value. | Management. Classification: LCC QA76.9.B45 (ebook) | LCC QA76.9.B45 M34 2019   (print) | DDC 005.7--dc23 LC record available at https://lccn.loc.gov/2018022753 ISBN: 9781138593060 (hbk) ISBN: 9781138593084 (pbk) ISBN: 9780429489679 (ebk) Typeset in Bembo by Sunrise Setting Ltd, Brixham, UK

To dad, brothers, wife and son. Atal To those that matter most, my wife and our children. Mike

CONTENTS

Acknowledgements  Foreword by Tom Davenport 

ix x

  1 Introduction For the practising manager  2 A non-technical book  2 Structure of the book  3

1

  2

Big Data revolution Data driven decisions and value creation  5 History of data and Big Data  10 Data and analysis  14 Data analysis and statistics  14 Data analysis and computing  15 Google web search  16 Big Data analysis on the cloud  17 Structured data  17 Unstructured data  19 Big Data  22 Summary 27

5

  3

Creating value from Big Data Value drivers in commercial organizations  30 Market value and non-financial values  36 Investments where Big Data can create value  39 Summary 54

30

  4

Big Data techniques and solutions Big Data analytics  55 Data analytics techniques  60 Summary 85

55

viii  Contents

  5

Introducing the model: design and implementation C-ADAPT model of Big Data value creation  89 C-ADAPT worksheet  105 Summary 121

87

  6

Big Data case studies Ooredoo (formerly Qtel)  124 Domino’s Pizza  126 Leading antivirus company  127 Gate Gourmet  129 Tesco 130 Delta Airlines  132 Intel 133 TXU Energy  135 OmedaRx 138 John Deere  140 Airbnb 141 Walmart 143 Huffington Post  145 Summary 146

123

  7

What practitioners say Big Data is important – very important!  149 Key value from Big Data  151 Challenges in implementing Big Data projects  155 Summary 156

148

  8 Conclusion and discussion

157

Index 

160

ACKNOWLEDGEMENTS

The completion of this book could not have been possible without the participation and assistance of so many people whose names may not all be enumerated. Their contributions are sincerely appreciated and greatly acknowledged. However, we would like to express our deepest appreciation and indebtedness, particularly to the following: Our colleagues at Ashridge Executive Education at Hult International Business School for their endless support during market analysis and research. Special thanks go to Steve Seymour and Vicki Culpin from Ashridge. We would also like to thank Abhishek Lakkar and his team for image design and Swati Suramya for her valuable contribution in editing this book. We thank Devashish Bharti, Matej Mišík and Paul J.J. Payack, who have participated in our interviews and Mats Abrahamsson, Dr Christopher Ahlberg, Manoj  Gupta, Mahendra K. Upadhyay and Amit Chandak for spending time and reading the unedited version and providing first reviews on this book. We are also thankful to Lucy McClune, Judith Lorton, Alex Atkinson and all the Routledge team for their very professional and efficient project management in publishing this book. We would like to especially thank Tom Davenport, who has written the foreword for this book and set the perfect scene for the reader. Last but not the least, thank you to all the readers, students, technologists and managers who, we hope, will learn and apply what we have presented in this book. Thank you! Atal Malviya Mike Malmgren

FOREWORD

More than a decade ago, Jeanne Harris and I co-authored a book called Competing on Analytics. One topic that we discussed often in researching and writing the book was whether all industries could or desired to move toward analytics-based capabilities. I believe I said at the time that it seemed unlikely that the fashion industry would ever become highly analytical. It seemed the paragon of an intuition-based industry, and I wasn’t sure whether there would ever be enough good data to analyze in any case. Suffice it to say that I was wrong. The fashion industry has moved heavily into analytics and big data. A quick Google search for ‘fashion analytics’ identifies such content as ‘7 novel ways of using big data in fashion’ and the ‘top 5 analytics trends in fashion retail’. A once-intuitive industry has clearly embraced data and analytics as a primary means of making decisions. That industry is hardly alone in its transformation. The authors of this book describe ‘big data’ as a particular kind of data that’s particularly large in volume and relatively unstructured. I support that definition, but I think big data has actually become a much broader concept. It has become the way we live now. There should be no doubt, as the authors demonstrate with many examples in this book, that data has penetrated virtually every industry and realm of human activity. It dominates sports, politics, social media, health care and most business enterprises. So every organization now has access to more data than ever before. In order to make sense of that data, we have to analyze it, determining patterns and trends. Whether one calls that collection of activities ‘big data’ (as do the authors), ‘business intelligence’, ‘analytics’, ‘machine learning’ or ‘blue bananas’, they are an essential element of life in current economies and societies. The terms change, but the activities themselves remain remarkably similar over time – at least over recent decades. This shift from a relatively narrow technical concept to a social and economic movement has a number of implications. It means, for example, that big data is no longer a fleeting trend. In the early days of the idea, companies embraced big data projects simply because they felt the need to try them

Foreword  xi

out. ‘Proofs of concept’ and prototypes abounded, but few projects yielded production applications or led to new business processes. At the core of this book, however, is the notion that big data projects are becoming a mainstream business activity. And like any other form of business initiative, they have to enable value substantially in excess of their cost in order to be worth doing. The sections in this book about how to create and measure value will be helpful to anyone who is attempting to generate economic returns from big data investments. Early big data projects were typically selected in an idiosyncratic fashion – someone in the organization had a particular passion for an issue, or a manager with clout bankrolled a project that benefitted his/her unit or function. But that’s not a good long-term basis for big data project decision-making. Hence another particularly useful aspect of the book is the C-ADAPT model, in which a systematic process is used to identify, scope, develop and test quantitative analyses within a business or organization. The model is explained though the use of a detailed example. Employing such a process can help to ensure that big data projects are framed and implemented in a way that will bring value to the organization. Big data has also had a reputation for being difficult and esoteric. But this book goes a long way toward dispelling that myth. Chapter 4 of the book is devoted to ‘Big Data Techniques and Solutions,’ and the fact that a great deal of useful knowledge about the subject can be packaged up in a single chapter is evidence of the accessibility of big data analysis. This book most impresses me, however, not because of any particular content or topic, but because of its practicality and usability. I am a big fan of collaborations between academics and practitioners, and this is one of those. Ashridge and Hult have always focused on business research and writing that is usable by practising managers and professionals, and Big Data for Managers clearly fits into that tradition. One clear indication that the book is intended for people who prefer specifics to abstractions is the astounding number of case studies in Chapter 6. Another is the set of survey results and interviews with practitioners in Chapter 7. If your objective, then, is to acquire practical knowledge on how to analyze big (or small, for that matter) data for business results or organizational value, you have come to the right place. You’ll find frameworks, methods, guidelines and examples that will all make the process easier. As they say on Star Trek, ‘resistance is futile’. The world is moving to big data, and you should join the movement lest it leave you behind. Thomas H. Davenport Distinguished Professor, Babson College, and Fellow, MIT Initiative on the Digital Economy; co-author of Competing on Analytics, Big Data @ Work and Only Humans Need Apply

1 INTRODUCTION

Creating value is the most fundamental purpose of any business. Though this value can be financial or non-financial, based on the business and project you are working on, the sole purpose of most businesses is to create financial value for shareholders. Businesses use various innovative ways and technology to create this value and in the list of these technologies or instruments, Big Data is the latest addition. In this book, the aim is to cut through the hype around Big Data, understand its less technical and more business-related aspect, but also inform and support those millions of managers that face pressure to invest in and make decisions around Big Data for their organization. Big Data is about technology and how technology has enabled information to be gathered at an unprecedented scale. The good news is that this information source can potentially offer an advantage over the competition or help serve your customers better. However, taking advantage of the technical developments in data and information handling requires investments in money, resources and time. For many business managers, this fast-moving technical environment poses the challenge of knowing where and why an investment in Big Data is justified and can make a return. To make an investment decision in business you need to understand the underlying drivers for how value is created, and in the context of Big Data what will be the likely outcome of your decisions. It may be possible that you and your team can start and run the most successful Big Data project to achieve a specific goal or address a challenge, but it is not always the case that you are creating short-term or long-term value for the company. So, with this book we are setting out a very practical approach and context for managers who have already started or are thinking about starting the journey of a Big Data project. We provide tools, techniques and processes for executing Big Data projects on one hand and value-creating process and measurement on the other hand – and we want managers to read, learn and practise both aspects of the

2  Introduction

businesses so you will make appropriate business decisions for making longterm sustainable value for the company.

For the practising manager From first-hand experience of working with practising managers in different organizations of varied sizes, we have realized that while technologies and inventions are highly valuable for businesses, often non-technical business managers face challenges in understanding the true potential of such technologies and using them for the organization at the right time. At the same time, the web, mobile and social networks and other platforms are producing enormous amounts of unstructured data, which can hold a wealth of market intelligence. Unlock this ‘Big Data’ and you have the power to build sustainable value in several forms. The purpose of this book is to help managers understand how and in what ways Big Data can be used to generate more revenue, save costs and come up with innovative offerings – but on top of everything else, it can create sustainable value for businesses. It can help managers understand what Big Data is and what it is not. It can also help managers have robust conversations with those inside and outside the organization who propose that Big Data is the answer to business problems but are not specific enough about how their investment can create value. The purpose of the book is to enable readers and practising managers to understand the potential of Big Data, plan to execute projects where needed but more importantly assess the outcome to ensure that value is created for the organization. A major take-away is the business model and proprietary C-ADAPT framework that can be used to understand how and where we can execute Big Data projects and value is created using Big Data.

A non-technical book This is a book for managers and not a technical book for techies and IT professionals and students. It is important for managers to have a good overview, not least to be able to understand and discuss with IT professionals and consultants. This does not mean that we have not used some technical words or discussed the evolution of data generation and usage, however we have tried to keep this to a minimum. This book presents enough technical aspects of Big Data that should be picked up by those aspiring managers who are planning to take a plunge into this ocean of opportunity. We have also not covered Big Data hardware or

Introduction  3

storage in detail but have focused more on analytics and the intelligence part of Big Data from where most businesses create value.

Structure of the book Although this it not a technical book, Chapter 2 has two sections. The first discusses how some organizations apply Big Data technologies, often resulting in disruption in traditional markets.The second section provides a historical review of the development of data science and some of the technologies that have developed over time. This is an important section as it provides some of the basic knowledge of terminology and the technology that underpins Big Data. In particular, it gives an explanation of the difference between structured and unstructured data. The explosion in unstructured data is from the conversations, blogs and web pages on the internet that can hold important insights on customers, trends and sentiments expressed among millions of internet users. Mining this data is what Big Data is all about. Chapter 3 is a primer for the understanding of value creation. It explains how financial value is calculated and gives examples of how Big Data can be used to create value. As this book is aimed at managers in organizations, a good understanding of the drivers of value creation is essential for any investment decision including Big Data. The chapter is built around the financials of a typical mid-sized business and will use its pro-forma income statement and balance sheet to show the potential impact of investment in Big Data. The aim of this chapter is a good understanding of the drivers of value creation. We expect managers to implement their learning from this chapter in measuring the success of any future project that they may run. We have also suggested a model and framework in Chapter 5, where value creation will be tested and acted on, in the last stage of the model. Chapter 4 is a description of Big Data technologies. This chapter takes us to the next stage – understanding the data and impact on value creation is associated with the analysis and extraction of insights from the data collected. In this chapter, we will be discussing different techniques of data analytics from old statistical models to the latest predictive analytics and data visualizations. The chapter will get technical at times, but this is inevitable, given that Big Data is underpinned by technology. Our aim is to give an overview and to provide lists of the pros and cons, so that a manager can have an excellent quality discussion with data scientists or technical teams. An important part of Big Data is unstructured data such as emails, blogs and text-based data and the chapter gives an overview of the different analytics techniques that can be used for this type of analysis as well. Chapter 5 is the key take-away of this book. Based on our work and observations from many Big Data studies, we present you with a model that

4  Introduction

can help any practising manager to lead Big Data projects. In Chapter 5 the C-ADAPT model of Big Data value creation offers a systematic model and practical template for managers who are tasked with building strategies for Big Data projects. We introduce the model and framework and explain the different elements and how they are used. The last stage of the model will help you measure the value created to ensure that the project is a sound investment in time and effort or suggest how you can make changes in the next iteration of the project. We also present the main tool to use with the suggested model – the C-ADAPT worksheet – which will help you during the process of executing the project and will keep you focused and help you drive the project accordingly. Chapter 6 includes a range of case studies around Big Data analytics and value creation. The cases are from technology companies like Intel to agriculture-​related companies like John Deere. The reason for including such a variety of cases is to show the reader that Big Data analytics can be used across any industry and different types of businesses. Fundamentally in all the cases, you can see that the C-ADAPT model can be mapped or applied if needed. The authors worked with first-hand experience in some of the cases, but other cases are contributions by other practitioners in different industries. Our intention is to present these cases, so you can identify your own business problem where possible. These cases are also written in a very concise and direct manner – if more details are needed or you would like to discuss anything, then you can reach out to the authors. The concluding chapter discusses the overall take-away and some of the issues we can see over the horizon on a fast-moving and disruptive technological development that most business managers will need to get their heads around and make decisions about. If you want to keep your reading light, you can skim-read Chapter 4 – Big Data techniques and solutions. Also, if you feel you are well versed with the concept and process of creating value, then you can skim-read Chapter 3. When you are done with reading this book, we would hope that you will be more confident about the subject and process of value creation.

2 BIG DATA REVOLUTION

Data driven decisions and value creation If you are in a business that produces products or services that no other business does, there are no substitutes for your offerings in the market, there is a growing demand for your offerings and your customers know how to reach out to you, then your business is in a very advantageous position. Mostly this kind of business can be categorized as a monopoly business and the only thing you can hope for is that no other business will start to produce a similar or substitute product in the future. In today’s world, monopoly firms can be formed only under very special circumstances – some examples may be when the government grants a special monopoly status to a firm such as the Post Office,1 but this is not the case with most of the businesses around. For most businesses getting to this advantageous position can seem very unrealistic and difficult, and even if a business is in such a situation, it is very unlikely that it can continue under the same conditions for long. Competition is the key component of any market – in this world, it is very rare to find any business without competition. Businesses try to create entry barriers by using unique ways and strategies to keep their competitors away from the market they are operating in. Sometime such strategies are also driven by their positioning or market segment. In a healthy competitive environment, competition plays the role of catalyst for innovation and growth in the market. In the UK, supermarket chains Aldi and Lidl are known for their budget offerings and they attract price sensitive customers to their stores. In the fourth quarter of 2015 these two supermarkets demonstrated sales growth of 13.3% and 18.5% respectively.2 At the same time Waitrose, a supermarket with premium price products, has seen the worst dip in their sales since 2006.3 Then in 2016, Waitrose became innovative and launched a new discount campaign – a ‘Pick your own offer’ where they decided to offer 20% discount on the most frequently used 10 products from their selection of

6  Big Data revolution

100 products. This strategy worked very well and the new campaign boosted their sales by 3.7% while other supermarkets like Tesco, Asda and Morrisons faced decline.4 Waitrose’s discount strategy was inspired by the growth of the budget supermarkets’ success and post-recession market sentiments that have affected the buying capacity of the average customer. Coming up with the right discount strategy was fuelled by extensive market research that included market and competition data and user behaviour data analysis and business intelligence. For Waitrose, their ‘Pick your own offer’ was one of the most successful strategies that helped them in securing an initial 700,000 customers within three months of its launch.5 This is one of the many examples where retail businesses have recorded sales data and using analytics have established customer journey and buying behaviours of their customers with a great degree of accuracy. Tesco, a large retail chain in the UK, is known for their introduction of a loyalty card (Tesco Club Card) – they use customer spending and usage data in establishing buying patterns and consumer behaviour of their customers. In 2008, Brian Chesky and Joe Gebbia  launched a company called Airbnb. Airbnb is an open marketplace that enables people to list, find and rent vacation homes/spare rooms for a fee. Airbnb challenged the hotel industry that had been in existence for hundreds of years. For well established players such as Marriott, the key value of the business is in the fixed assets and their business is driven by the size and location of the properties they own. Traditionally the hotel industry was assumed to be a business of heavy capital investments because of the real estate costs involved. As we know, in many cases, innovation comes from other industries. A new era in the hotel industry was foreseen by technology companies. Airbnb and many other similar companies like OYO Rooms and Zoho Rooms have approached the problem from a completely different angle. They realized that there are many travellers and small business professionals who cannot afford a room in a decent hotel with the right location at short notice and at the same time, there are thousands of residents, in the same area, living in their houses with some spare room, that they can rent out to travellers. Airbnb started to work on this problem from the root and created ways for room owners and travellers to connect and do business together. Keeping assets light was a key strength of the company and one of the main reasons behind their tremendous growth in a short space of time. When Marriott announced in 2014 that they were planning to add 30,000 rooms in a year, Chesky tweeted that Airbnb could add 30,000 rooms in their inventory in the next two weeks. By spring 2014 Airbnb had 550,000 properties listed on their website with 10 million registered guests.6 What Airbnb has done in the hotel industry is the same as what Uber has done in the taxi industry.

Big Data revolution  7

In 2008, on a snowy day, Travis Kalanick was not able to get a taxi to go to a conference and, out of this situation, Kalanick realized the need for a taxi booking service through your mobile in just two taps. Then, at the LeWeb tech conference, he had a fateful conversation with StumbleUpon founder Garrett Camp. Camp told him about his idea for a luxury car service that was convenient and didn’t cost $800 for a ride, a price he had once paid. Realizing that this was an issue faced by countless travellers around the world, with Garrett Camp, Kalanick launched Uber Cabs and that is what is now known as Uber.7 Uber is not a taxi company, but a technology solution to a problem that was considered as a transportation related problem. Travellers can book an Uber cab through a mobile application available on Google  Play Store as well as the App Store.  When the user’s GPS is turned on, Uber notifies the drivers who are near the location, thereby ensuring that the customer gets a cab as soon as possible at an accurate pickup time. Uber is now available in many countries across the world, including the United States, United Kingdom, India, France and Australia. Uber has faced several instances of opposition from existing taxi companies and other organizations, but their innovative solution was always supported by millions of users who were giving business to Uber. As of September 2016, Uber gets on average 1 million rides every day and adds 50,000 new drivers as their partners every month.8 Airbnb and Uber are two examples of innovative businesses where these new entrants have changed the way the existing companies were doing business in the industry. By no means we can assume that Uber will replace all taxi companies or that Airbnb will outpace the hotel industry, but what these two companies have done is find an innovative way of solving a problem for users of the service and fortunately there are millions of such users ready to use their services. In reality, all businesses need to know is how to deliver faster, better or cheaper solutions to their customers’ needs than their competitors do. To achieve this, businesses need to make strategic choices to be in a leading position in the Industry. Bruce Henderson, the founder of Boston Consulting Group9 and Michael Porter,10 professor at Harvard Business School, are two known thought leaders who have introduced the idea of business strategy to the business world between 1965 and 1980. We will not go into details of their notable contributions here, as you can easily find their literature on the Internet. For the likes of Henderson and Porter, the basic idea of strategic thinking in business comes from a deep understanding of different components and stakeholders of any business (processes, products and resources) and changing them to produce a better outcome (value) and create an advantageous position.These changes led by business understanding come from insightful information about your business, competition and industry – collectively known as business intelligence. Business intelligence is often and

8  Big Data revolution

more recently considered to be a technical area of business where several reports and graphs help managers in making strategic decisions. In reality, it has nothing to do with technology except that now we have ways of generating more complex information that can be insightful for businesses. In 1865, Richard Millar Devens in the Cyclopaedia of Commercial and Business Anecdotes coined the term ‘business intelligence’. Devens mentioned business intelligence to describe how Sir Henry Furnese, a banker, created value by receiving information before any of his competitors and by acting on that information in good time.11 The world of business has changed a lot in the last 150 years. Especially in the last 20–30 years technological innovations have empowered business leaders to take important and critical decisions quickly and effectively based on solid proof (data) points. Now business leaders have access to more information and intelligence than ever before and there is little room for assumptions and guessing. Businesses and customers are now producing more data than ever before, and businesses have resources to store, use and analyze this huge amount of data quickly. Many technology giants like Oracle and SAP have been helping businesses in building business intelligence (BI) suites as part of their basic systems, and their business intelligence systems are helping businesses in making smart choices and informed decisions. For example, retailers such as Tesco in the UK and Walmart in the USA have moved beyond relying on a ‘reduce the price and increase the sales strategy’ for all their customers, and now they have started to record and analyze the spending behaviour of their customers so they can send very targeted offers to those customers who are more likely to buy specific products. If you do online shopping from Tesco or use your Tesco club card, for which you get incentives, then you will start to receive discount vouchers for the products you use more, or you may want to use them in future. Tesco not only knows how many milk bottles your household needs in a week, it also knows what kind of new product you may be interested in too – and this is done by intelligent user-profiling algorithms based on their business intelligence suite. Many websites are now using data to develop their recommendation engine, so they can offer different products to different customers based on their preferences. Amazon.com is well known for their recommendation engine – if you buy a recipe book from Amazon.com, you will likely see similar cook book recommendations on the website and in your email newsletters. At the root is the retail giant’s recommendation system that they call ‘item to item collaborative filtering’, which is based on a number of simple elements of information such as ‘what a user has bought in the past’, ‘which items they have in their virtual shopping cart’, ‘items they’ve rated and liked’ and ‘what other customers have viewed and purchased’ etc.  Behavioural recommendations can also be found on other web portals like YouTube and LinkedIn.

Big Data revolution  9

Data driven decisions have also helped businesses in making disruptive market changes. For example Myntra.com, a leading e-commerce startup in India announced that they would be shutting down their web portal on 1 May 2015 and would continue their business from mobile apps only.12 Eighty per cent of Myntra.com’s traffic and 70% of sales had been recorded from mobile devices in the previous year. One of the main reasons for this was that their customers had easy access to low cost internet through their mobiles. There are many cities and villages that have no access to decent internet access through wired connections and mobile phones are filling this gap. India is one of those countries that have skipped the big age of landline phones and wired internet connections because of the introduction of low cost mobiles and a competitive service providers market leading to the provision of very low-cost call and data services. On the basis of these facts and related intelligence coming from Myntra.com’s post sales department, management decided to move ahead with its mobile-only strategy to focus on the fast-growing Indian mobile customer base which has seen multiple increases in the last 2–3 years. There are many examples where data-based decisions have helped companies to grow really fast. Companies like Uber or Myntra.com have moved one step up in their technological offerings. But there are also some examples where companies couldn’t ‘read’ the right information coming from the data and ended up taking decisions that have resulted in loss. Morrisons-Kiddicare is one such example of huge losses. Morrisons is the third largest supermarket chain in the UK and in 2011 Morrisons acquired an online maternity store, Kiddicare, for £70 million, with the expectation of making a multi-channel sales opportunity by creating online and offline stores running simultaneously. What Morrisons missed was that online and mobile shopping trends in this area were growing fast, with higher smartphone penetration rates in the market and thus going offline could be a costly decision. Morrisons acquired the failed Best Buy stores in the UK and converted them to Kiddicare stores. From 2011 to 2014 Morrisons faced huge losses and finally in 2014 they sold Kiddicare to private equity firm Endless for £2 million and booked £163 million as an overall loss in this whole episode.13 Now Kiddicare is back, operating as an online store. The discussion around data and analytics is not just limited to big and old organizations; with the help of many free or low-cost analytics tools, and reduced cost of data storage through cloud computing, small and medium-sized businesses have started taking advantage of data analytics in their daily operations. Tools like Google Analytics are must-have tools for most businesses with an online presence. Many technology startups are coming up with more intelligent and low-cost analytics tools and even the

10  Big Data revolution

analytics tools market is becoming very competitive. Just as an example, in the span of 5 to 7 years, many social media analytics companies have launched their solutions and established their position very well among small- and medium-sized businesses. Opportunity and market competitiveness has started to result in consolidation within this space. A popular social media analytics company Hootsuite has raise more than $246 million and acquired more than 10 businesses in last six years – including its competitor companies Seesmic and UberVU in 2012 and 2014 respectively.14 Data driven decisions are not only used by businesses in selling more products or services but are also being used in multiple areas from improving business operations and resource planning to product development and innovations. In the last 10 years, customers, businesses and devices have produced more data than ever before. Based on research results produced by IBM in 2012, 90% of the world’s data was produced in 2011 and 201215 – so businesses can only assume that there is more data to learn, act on and improve in this digital world.This data is being generated very fast in huge quantities and is coming in different formats from different sources including sensors, social media, mobile devices, online activities, log files and many other sources, and this presents a huge opportunity for the business manager and executive who can take advantage of these ever-increasing data sets. ‘Big Data’ is the term used for this fast growing, extensive and diverse data that is too big to store in an old-style single server, too unstructured to be analyzed by old-style BI systems and too fast to hold in a static data warehouse as was the case earlier.16 It may be of limited value for business managers, as they may not understand the true value of this new phenomena. Form storage to analysis, businesses can save huge costs, generate revenue and extract valuable actionable insights. Big Data is a key instrument for managers to use in crafting and implementing winning strategies.

History of data and Big Data Any enterprise CEO really ought to be able to ask a question that involves connecting data across the organization, be able to run a company effectively, and especially to be able to respond to unexpected events. Most organizations are missing this ability to connect all the data together. – Tim Berners-Lee (Inventor of the World Wide Web)

The problem of managing data coming in huge volumes from various sources and extracting value from that is not new. In 1944, the Librarian of Wesleyan University, Fremont Rider, had suggested that the size of libraries of American Universities was doubling every sixteen years. He also predicted that by 2040, Yale University library will have approximately 200 million volumes, which will occupy over 6,000 miles of shelves.17 Clearly, Rider was talking

Big Data revolution  11

about the challenges of storing, managing and using data that is more than the available data storage capacity. At that time, this was never classified as a Big Data problem, but in that context, Rider was referring to similar phenomena that today can be correlated with Big Data. From the time of Rider, we have seen many developments in the field of technology and particularly in the field of data management. For a long time, data were stored and managed in a smart way within the reach of Data Base Management Systems (DBMS) and then Relational Data Base Management Systems (RDBMS). DBMS was invented in the early 1960s and RDBMS in the 1970s – both in the labs at IBM.18 Working with relatively small systems, we were satisfied with DBMS or RDBMS approaches and intelligent systems built on them, until we stumbled upon the phenomenon of the Internet. Born in 1955 and having studied at Oxford University, Tim Berners-Lee was working with CERN in 1985 when he published a paper titled ‘Information Management: A Proposal’. In this paper Berners-Lee presented a system of creating, sharing and distributing information globally and named it the World Wide Web.19 Little did Berners-Lee know at the time that this new system would open opportunities for the whole world to create and share information in huge volumes and that it would be challenging to store and manage the data generated in this process in the existing systems. Industry leaders should have seen this point as an early start of Big Data era, but it was not until early 2000 when two leading companies in the technology world – Yahoo! and Google – identified and predicted the issues of storing and managing huge (or Big) data issues and initiated work on finding a solution for the same. Their work was mainly motivated by their own needs of managing webpages and search data that was always testing the limits of the existing infrastructure and processes. In 2013 Roger Magoulas defined Big Data – when the size and performance of the data becomes part of the problem.20 Big Data was not just limited by the definition that Magoulas suggested. Fast growing, huge and varied data has made exiting storage and old-style analytical systems redundant and that was a really big problem (and opportunity). Storing and using this huge amount of data was the first problem that needed to be addressed before moving ahead with identifying other opportunities that the data might present for businesses. Though the growing size of data might be a big problem for many businesses and some big companies have resources to manage this data storage problem, a few were working on solving it in innovative ways and were focusing on its next big issue – data retrieval and management. Companies managing millions of webpages and trying to index them in their databases in real time were the real force behind the invention of Big Data technologies. Google and Yahoo! contributed a lot in the early days of the development of these technologies.

12  Big Data revolution

In 2004, two Google employees – Jeffery Dean and Sanjey Ghemawat – presented a white paper defining a programming model called MapReduce. For around five years many Google employees including Dean and Ghemawat were working on computing large amounts of data coming from internet users’ activities such as number of pages crawled per host, the set of most frequent queries in each day, etc.They could distribute large data sets on multiple servers for parallel processing and could use the MapReduce model to process these large data sets in considerably very short times.21 This was considered a big win at that stage. But around the same time, engineers at Yahoo! were working on solving the same problem in a different way. The name of the Yahoo! project was Nutch, and it was led by Internet Archive Search Director, Doug Cutting, and a graduate student of the University of Washington, Mike Cafarella. Working on Nutch for several months and trying to manage around a billion web pages’ data, Doug Cutting came across Google’s white paper that mentioned how MapReduce solves a similar problem in a better way. Inspired by the MapReduce framework, Doug Cutting and Mike Cafarella implemented this framework on an open sources technology and launched Hadoop in 2005.

Apache Hadoop is an open-source software framework for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware. In the field of data storage, the fact cannot be avoided that hardware failures are common problems and if possible, these failures should be handled automatically. Hadoop has solved this long existing problem by its design, where all the modules were designed to automatically handle failures by the framework. The core of Apache Hadoop consists of a storage part, known as Hadoop Distributed File System (HDFS), and a processing part called MapReduce. Hadoop splits files into large blocks and distributes them across nodes in a cluster. To process data, Hadoop transfers packaged code for nodes to process in parallel based on the data that needs to be processed. This approach takes advantage of data locality – nodes manipulating the data they have access to – to allow the dataset to be processed faster and more efficiently than it would be in a more conventional supercomputer architecture that relies on a parallel file system where computation and data are distributed via high-speed networking. The base Apache Hadoop framework is composed of the following modules: 1.  Hadoop Common – contains libraries and utilities needed by other Hadoop modules; 2.  Hadoop Distributed File System (HDFS) – a distributed file-system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster;

Big Data revolution  13

3.  Hadoop YARN – a resource-management platform responsible for managing computing resources in clusters and using them for scheduling of users’ applications; and 4.  Hadoop MapReduce – an implementation of the MapReduce programming model for large scale data processing. The Hadoop framework itself is mostly written in the Java programming language, with some native code in C and command line utilities written as shell scripts. Though MapReduce Java code is common, any programming language can be used with ‘Hadoop Streaming’ to implement the ‘map’ and ‘reduce’ parts of the user’s program. Other projects in the Hadoop ecosystem expose richer user interfaces. The term Hadoop has come to refer not just to the base modules above, but also to the ecosystem, or collection of additional software packages that can be installed on top of or alongside Hadoop, such as Apache Pig, Apache Hive, Apache HBase, Apache Phoenix, Apache Spark, Apache ZooKeeper, Cloudera Impala, Apache Flume, Apache Sqoop, Apache Oozie, Apache Storm. Apache Hadoop’s MapReduce and HDFS components were inspired by Google papers on their MapReduce and Google File System.

Many companies and startups have created their own versions of Hadoop using Apache open source framework. On data retrieval and management front, many companies have invented their own query or scripting languages – such as Pig and Hive were invented to help businesses analyze this huge amount of data stored in Hadoop nodes.22 In today’s data-rich world, businesses are collecting huge amounts of data from diverse sources such as social media, web traffic, system log files, customer feedback, employees’ activities etc. – so now we have access to a lot of data from almost all parts of our business and this presents a huge opportunity for businesses if the right value can be extracted and derived using the right data – which we will discuss in coming chapters. Recently we have seen many cases where involvement and analysis of data has changed the scenario completely, whether it is business or politics.The first Big Data election – the US presidential election of 2012 – is one such example. In 2012,Team Obama was clearly the winner, not only in winning the election but also in understanding the changing behaviour of voters with the development of social media and mobile applications.The Obama team understood the importance of execution and the difficulties of data complexities. One of the first priorities before the election was to undertake a massive, 18-monthlong database merge so all the data could be housed in a single repository. The database focus also allowed the Obama camp to think expansively in its

14  Big Data revolution

approach to metrics. ‘We are going to measure every single thing in this campaign,’ campaign manager Jim Messina told newspaper Time in an interview.23 Messina was also given reliable resources to work with, and hired an analytics department five times bigger than the 2008 operation. It was a very different and contrary approach to that taken by the opposition – the Romney camp. Based on the information shared by the Obama team, they were able to run simulated elections over 66,000 times every night, based on the data collected on that day, to ensure that they were observing and noting the impact of each and every data point associated with the voters and elections as a whole. In addition, the demographic information they collected and scored against other factors allowed them to find more targeted ways to buy television advertising to reach their ‘micro-targeted’ voters. Clearly this strategy resulted in favour of Obama in the first Big Data election ever. Before handing over the baton to Donald Trump, Obama won the election twice and ran the nation for eight years. In both of his elections, Big Data analytics added huge value.

Data and analysis You can have data without information, but you cannot have information without data. – Daniel Keys Moran (Computer programmer and science fiction writer)

Clearly data is all around us, we observe it, use it, ignore it or don’t even know about it (unless you are living in a remote place without any connection with the outside world), but it is not a new ‘thing’ at all. Long before the time when human beings started to communicate properly, using sign language or less developed vocal variations that were used to identify difference activities or object, data existed around us. As Thomas Davenport and D. J. Patil wrote in an article titled ‘Data Scientist: The Sexiest Job of the 21st Century’, in 2012,24 data analysis is getting a lot of attention nowadays and this is one of the jobs that is in high demand. It’s taking on an increasingly larger role in all sizes of companies, including small startups.The practice of data analysis has gradually developed over time, gaining huge benefits from the evolution in computing.The fundamentals of data analytics have been the same over many years through the processes of handling and dealing with data and performing analytics on them.

Data analysis and statistics Data analysis is rooted in statistics, which has a pretty long history. It is said that the beginning of statistics was marked in ancient Egypt, when Egypt was taking a periodic census for the building of pyramids. Throughout history,

Big Data revolution  15

statistics has played an important role for governments all across the world, for the creation of censuses, which were used for various governmental planning activities (including, of course, taxation). Similarly, businesses have been using data and statistics, directly or indirectly, for a really long time. With the data collected, we can move on to the next step, which is the analysis of that data. Data analysis is a process that begins with retrieving data from various sources, and then analyzing it with the goal of discovering beneficial information. For example, the analysis of population growth by district can help governments determine the number of hospitals that would be needed in each area. Same information about the population growth can help businesses to expand or shirk their operations in different areas.

Data analysis and computing Advances in collection mechanisms The invention of computers and the subsequent advances in computing technology have dramatically enhanced what we can do with data analysis. When the process of data collection was manual (without computers), it was a long, tedious and error-prone process. In 1880, in order to take the census in the US, it took a total of 7 years for the collection of data and to present the final report. In order to shorten the time it took to create the census, in 1890, Herman Hollerith invented the ‘Tabulating Machine’. This machine was capable of systematically processing data recorded on punch cards. With the help of the Tabulating Machine, in 1890, the same process (census) could be finished in only 18 months and that too at a fraction of the original cost. Relational databases The turning point in the field of data collection and analysis can be considered the appearance of RDB (relational database) in the 1980s which allowed users to write query language (SQL) to retrieve data from an existing database. This combination of RDB and SQL empowered users to start running and analyzing on-demand queries, based on business needs and that way, business rule changes could be incorporated very easily. It made the process of getting data easy and helped to spread database use. As you can see, the combination of easier and cheaper data collection with cheaper and faster data storage and retrieval technology has pushed the boundaries of what we can do with data. Data warehouse and business intelligence By the late 1980s, the amount of data collected was increasing fast and the cost of disk drives for storing this data was also increasing at the same rate.

16  Big Data revolution

This was a big challenge for businesses who were convinced of the value of the data but for whom the increasing size and cost was a continuous challenge. Then William H. Inmon proposed a concept called Data Warehouse – a system that optimizes the reporting and analysis on the basis of data time, operation type and time stamp associated with the data – this was different from the traditional data storage where it was a flat linear storage and retrieval process in most cases and data was stored with a timestamp, and operations such as DELETE and UPDATE were used much less frequently. So, if the sales department of any business wants to compare sales data trends for every quarter, data can be stored on a quarterly basis with the time stamp associated with it and can be queried on the basis of time stamp. Also, the term ‘BI (Business Intelligence)’ was proposed by Howard Dresner at Gartner in 1989. BI supports better business decision making through searching, collecting and analyzing accumulated data in business. The birth of the concept was only natural, given the quality of the technologies like databases and data warehouses available to support it. Big companies especially embraced BI by analyzing customer data systematically when making business decisions. It has taken some time for small businesses to adopt the concept of BI in its original form but now, irrespective of the size of the business, data collection and business intelligence are an integral part of it. Data mining Data mining is a computation process of identifying the patterns in a data set. This concept was introduced in the 1990s. By analyzing data in a different way from the usual methods, unexpected but beneficial results could be achieved. The development of data mining was made possible thanks to database and data warehouse technologies, which enable companies to store more data and still analyze it in a reasonable manner. A general business trend emerged, where companies started to ‘predict’ customers’ potential needs based on analysis of historical purchasing patterns.

Google web search The next momentous change was the Internet. In response to the demand to search a website on the web, Larry Page and Sergey Brin developed the Google search engine which processes and analyzes Big Data in distributed computers. Surprisingly the Google engine responds with the result which you mostly likely wanted to see in just a few seconds. The key points of this system are that it was ‘automated’, ‘scalable’ and ‘high performance’. A white paper on MapReduce in 2004 greatly inspired engineers, pulling in an influx of talent to respond to the challenge of handling Big Data. In the late

Big Data revolution  17

2000s, many open source software projects like Apache Hadoop and Apache Cassandra were created to take on this challenge.

Big Data analysis on the cloud In the early 2010s, Amazon Redshift, which is a data warehouse on the cloud, and Google BigQuery, which processes queries on thousands of Google servers, were released. Both came with a remarkable fall in costs and lowered the hurdles in processing Big Data. Nowadays, every company and even startups, which traditionally did not have a budget to conduct such analysis, are now able to repeat PDCA cycles (plan-do-check-adjust cycles) rapidly by using Big Data tools. Let’s think about the evolution of Data – structured, unstructured (semi-structured, meta data) and Big Data – in my opinion these are concepts that have existed around us for many years, and it is just that we are now able to realize, record and use them from the last few decades after the innovation of new technologies and systems.

Structured data Structured data refers to kinds of data with a high level of organisation, such as information in a relational database.When information is highly structured and predictable, search query results can more easily organize and display it in creative ways. Structured data first depends on creating a data model – a model of the types of business data that will be recorded and how they will be stored, processed and accessed.This includes the field, type or any restrictions associated with the collected data based on pre-defined business requirements or business rules. Data can be numeric, currency, alphabetic, name, date, address and any restrictions on the data can be restricted to certain terms such as Mr., Ms. or Dr. etc. Structured data has the advantage of being easily entered, stored, queried and analyzed. At one time, because of the high cost and performance limitations of storage, memory and processing, relational databases and spreadsheets using structured data were the only means to effectively manage data. Anything that couldn’t fit into a tightly organized structure would have to be stored on paper in a filing cabinet. A programming language, SQL (Structured Query Language), was created for managing and querying structured data in relational database management systems. It was originally developed by IBM in the early 1970s and later developed commercially by Relational Software, Inc. (now Oracle Corporation). Structured data was a huge improvement over strictly paper-based unstructured systems, but life doesn’t always fit into neat little boxes. As a result, structured data always had to be supplemented by paper or microfilm

18  Big Data revolution

storage. As technology performance has continued to improve, and prices have dropped, it was possible to bring into computing systems unstructured and semi-structured data. Retail banks are one of the oldest institutions in the modern business world. Before digitalizing their operations and using advanced systems to optimize their processes, banks were using big physical books and registers to record and manage data manually (i.e. by using pen and paper). Banks were managing different registers for different purposes including customer records, monetory transactions, loan accounts etc. and in most cases, the link among all of these books was a common number known as an account number, which was uniquely allocated to every customer. Not so long ago, retail banks used to give a paper passbook to their customers to keep track of all the activities on their bank account using the same account number – paper passbooks are still popular in some less developed parts of the world. Then came the time when bank operations moved from physical books to computers. Many processes were digitalized using the existing banking concepts – account numbers were used as a differentiating key (primary/ composite key) and different data tables were created to replace physical registers. Similar processes took place in other industries as well around the same period – 1970s to 1980s. Terms like DBMS, RDBMS, ER Diagram, ERP etc. were introduced and these concepts and technologies were widely used by different industries from the 1970s. From spreadsheets to ERP, typical enterprise data were

DBMS: A  database management system  (DBMS) is a computer software application that interacts with the user, other applications and the database itself to capture and analyze data. A general-purpose  DBMS  is designed to allow the definition, creation, querying, update and administration of databases. RDBMS: A relational database management system (RDBMS) is a database management system (DBMS) that is based on the relational model as invented by E. F. Codd, of IBM’s San Jose Research Laboratory. In 2016, many of the databases in widespread use are based on the relational database model. ER Diagram: An  entity-relationship diagram  (ERD) is a data modelling technique that graphically illustrates an information system’s entities and the relationships between those entities. An ERD is a conceptual and representational model of data used to represent the entity framework infrastructure. ERP: Enterprise resource planning (ERP) is a process by which a company (often a manufacturer) manages and integrates the important parts of its business. An ERP management information system integrates areas such as planning, purchasing, inventory, sales, marketing, finance and human resources. https://en.wikipedia.org/wiki/Relational_database_management_system

Big Data revolution  19

spread across different tables and tables were connected using relationship keys. So far data were stored and used in a predefined orderly fashion. Structured data refers to any data that resides in a fixed field within a record or file. This includes data contained in relational databases and spreadsheets. Data stored in pre-defined data-structures and similarly pre-configured data were mainly accessed and used by SQLs – structured query languages. SQLs were used for accessing logically spread data in different tables using common keys (or relations). Before 1995, structured data was easy to store and use because of its predefined known structure and relatively low volume. Companies like Oracle Corporation and IBM have provided platform, technologies and tools to access and analyze this data. Earlier it was safe to classify any data stored in tables and rows in a pre-defined orderly structure simply as data, which is now known as structured data to differentiate it from unstructured data, which we will discuss next. Structured data size, format and flow are always as per pre-defined expectations; SQL programming languages were the right choice for performing operations or analytics on structured data. Before we move ahead to discuss unstructured data, it will be worth mentioning that most of the data before the 1990s was created within organizations and was used by them internally. Sources of data have changed a lot and a big surge is being recorded in external data mainly generated by customers and their devices like mobile phones and usage of open social media portals like Facebook and Twitter.

Unstructured data Like structured data, unstructured data is also not a new concept for the world – do you remember how old newspapers were printed? Before the use of computers, newsprint was composed on manual machines by human hands working in reverse – manual typesetting – a daunting task if you think how easy printing is now. Unlike today, in those days, newspaper composers didn’t have any pre-defined structure of the page except the header of the page, which covers the name of the newspaper. Some days paper would be printed in two columns and some days in three columns, 5 blocks and 2 images. The structure of the newspaper was always changing based on the quantity, type of text and images. It was considered a big challenge for the typesetter to fit the data (news) in the most appropriate form on the basis of their structure on that day. Similarly, unstructured data comes in all shape and sizes, and it is challenging for anyone to store, analyze and use unstructured data the same as structured data. Where a database was introduced as a concept, most of us have focused on the part of the data that was easy to store and use, so unstructured data

20  Big Data revolution

was simply ‘ignored’ in the discussion. It was not that the unstructured data was not useful, but that the value generated from structured data was more sustainable in the given circumstances and technical capabilities. The inflow of unstructured data has increased since the late 1990s when more consumers have started to create data on the Internet – via blogs, forums, social media portals and on other places and devices. This new and innovative technology and behavioural change has presented a huge opportunity for organizations, alongside a challenge. Storing and analyzing the unstructured data might have been a possibility in the past as well as using the then existing technologies if the volume and speed of this data had not increased multiple times. Now most of us use social media and are connected to the Internet using multiple devices. When consumers post a message on Facebook, upload an image on Instagram or use any mobile application, their digital footprints are recorded and stored in various places. Apart from these, companies have started to store and use mobile usage data, digital sensor data and other log data in the hope of better understanding their customers’/consumers’ behaviours, market and opportunities. Many argue that the unstructured digital data that constitutes a big part of Big Data came into existence with Web 2.0 or the social media revolution – which cannot be considered as an entirely correct argument. Data with unexpected structure or unstructured data in digital form first started to pour in in the form of email text and other log files and then also in the 1990s when consumer discussion portals or forums came in existence. Multi-threaded conversations among different consumers were the initial point where this crowd-generated content creation started. Later, social media portals such as MySpace, Facebook, Twitter etc. provided a more intuitive way for a wider audience to create, discuss and share content of various forms including text, images, links and videos. At the same time the Internet and mobile technologies have advanced to reach out to a bigger part of the world which has accelerated the amount of unstructured data in multiple ways. According to estimates, up to 90 per cent of the data in any organization is unstructured. And the rate of growth of unstructured data in any organization is much higher than the rate of the growth of structured data. Apart from the content created by consumers on different social media and other sites, digital activities of consumers were also increasing a lot and these activities were continuously creating huge amounts of data for the companies behind the scenes – something like website click stream or mobile usage data that was helping businesses improve their offerings. But the real challenge was not just to handle the unstructured nature (variety) of this new data but it was the size (volume) and speed (velocity) by which

Big Data revolution  21

it was generated that was making the task more difficult. Thankfully Google and Yahoo! teams were able to foresee this challenge and were working on dealing with this challenge. Many organizations believe that their unstructured data stores include information that could help them make better business decisions. Unfortunately, it’s often very difficult to analyze unstructured data. Organizations can get help from many software solutions that read the unstructured text and present meaningful information in return. The primary benefit of these tools is the ability to glean actionable information that can help a business succeed in a competitive environment. Examples of ‘unstructured data’ may include books, journals, documents, metadata, health records, audio, video, analogue  data, images, files and  unstructured text such as the body of an e-mail message, web page or word-processor document. Because the volume of unstructured data is growing so rapidly, many enterprises also turn to technological solutions to help them better manage and store their unstructured data. These can include hardware or software solutions that enable them to make the most efficient use of their available storage space. Apart from unstructured and structured data, the third type is a mix of both – semi-structured data. Semi-structured data is information that doesn’t reside in a relational database but that does have some organizational properties that make it easier to analyze. Examples of semi-structured data might include XML documents and NoSQL databases. The term Big Data is closely associated with unstructured data. Big Data refers to extremely large datasets that are difficult to analyze with traditional tools. Big Data is a mix of all sorts of data structures, unstructured, semi-structured data or meta data, but an estimated 90 per cent of Big Data is unstructured data. Many of the tools designed to analyze Big Data can handle unstructured data. Unstructured data in this context also covers semi-structured and metadata. Semi-structured data is partially structured data such as email data – some parts like sender’s and receiver’s email address, time of email etc are pre-defined. They always follow the same format, but the content of the email can include simple text, links, images and different types of attachments. Similarly, Meta-Data that is data about data such as written within XML tags can also be partially preconfigured. We need not to go into detail about semi-structured and meta-data here, as much of that can be considered as unstructured data and fits into the theme of this book. More conventional industries such as agriculture, insurance etc. are also creating huge amounts of data that is then stored and analyzed to extract hidden insights.

22  Big Data revolution

Big Data ‘Big Data’ is the common term used for this fast growing, huge and varied data that is too big to store in old-style single servers, too unstructured to be analyzed by old-style BI systems and too fast to hold in a static data warehouse as was the case earlier. Big Data will be of limited value for businesses if they cannot analyze it and extract valuable actionable insights from this in order to craft and implement winning strategies. While the term ‘Big Data’ is relatively new, the act of gathering and storing copious amounts of information for eventual analysis is ages old. The concept gained momentum in the early 2000s when industry analyst Doug Laney articulated the now-mainstream definition of Big Data as the three Vs: Volume – Organizations collect data from a variety of sources, including business transactions, social media and information from sensor or machineto-machine data. Unlike in the past where storing huge data was an issue, modern technologies like Hadoop have reduced the resource cost and process load. Velocity – Data stream in at an unprecedented speed and must be dealt with in a timely manner. From mobile phones and RFID tags to sensors, internet and smart metering, there are many sources driving the need to deal with tonnes of data in near-real time. Variety – Data comes in all types of formats – from structured, numeric data in traditional databases to unstructured text documents, email, video, audio, stock ticker data and financial transactions. At some point two additional dimensions were also added to Big Data: Variability –  In addition to the increasing velocities and varieties of data, data flows can be highly inconsistent with periodic peaks. Is something trending in social media? Daily, seasonal and event-triggered peak data loads can be challenging to manage. Even more so with unstructured data. Complexity – Today’s data come from multiple sources, which makes it difficult to link, match, cleanse and transform data across systems. However, it’s necessary to connect and correlate relationships, hierarchies and multiple data linkages or your data can quickly spiral out of control. The amount of data that’s being created and stored on a global level is almost inconceivable, and it just keeps growing.That means there’s even more potential to glean key insights from business information – yet only a small percentage of data is analyzed. What does that mean for businesses? How can they make better use of the raw information that flows into their organizations every day? The importance of Big Data doesn’t revolve around how much data you have, but what you do with it.You can take data from any source and analyze it to find answers that enable 1) cost reductions, 2) time reductions, 3) new

Big Data revolution  23

product development and optimized offerings and 4) smart decision making. When you combine Big Data with high-powered analytics, you can accomplish business-related tasks such as: 1. Determining root causes of failures, issues and defects in near-real time. 2. Generating coupons at the point of sale based on the customer’s buying habits. 3. Recalculating entire risk portfolios in minutes. 4. Detecting fraudulent behaviour before it affects your organization. Most of the books and literature discussing Data and/or Analytics have missed the fundamental key difference between today’s new data (Big Data/ unstructured data) and yesterday’s old data (structured data) and here we are not talking about obvious attributes that we mentioned earlier. Basically, business leaders, managers and their technology teams have created this difference, long before data was created, stored and analyzed by businesses. The difference is control over the data sources – earlier businesses have created a closed, controlled environment of business processes and data collection to the extent that much before the data collection process is started, businesses know what kind of data or information they will be collecting and from where, and how they will be storing and processing them. System analysts have pre-defined things like ‘what data’, ‘from where’ and ‘in what format’, they will be collecting and analyzing it along with their structure, size and flow. In fact, items like the length of a data value are also defined in advance so that any input data should follow the same criteria as defined during system and database creation in the beginning. This is the process that businesses have been following for a long time, maybe because of technical limitations. Many businesses have created boundaries around themselves and they are mostly dealing with very familiar data sets coming from known sources in a closed environment. Many times, where pre-defined known data are the only data in the organization, executives face difficulties in identifying any key insight from the same defined data sets, and thus they limit the possibilities of innovating the business and gaining deeper knowledge of customer behaviours and business conditions. In such cases, managers are also highly dependent on the technical team to present insight into the business. We argue that this controlled data collection approach has imposed limitations on harnessing new opportunities, imaginations, innovations and value creation prospects. New data, as opposed to old data, has broken these boundaries, and businesses have started to receive (either by choice or not) huge amounts of data from sources, most of which are not in their control, and started to take a reactive approach to handling them. Some businesses have taken this

24  Big Data revolution

opportunity to innovate and make progress that others are still trying or thinking about. We now call this ‘new data’ as big (unstructured) data is not as well defined as old, structured data. And now as we are learning to deal with this new data coming from new sources we are also able to deal with unstructured data produced by our internal systems, such as system logs, most of which was ignored in our previous systems. Here are a few examples of the kind of new data available to collect now, which businesses are storing and analyzing in order to extract more value for all their stakeholders. Customer data Businesses can collect a lot of information directly or indirectly from their customers to improve their offerings – here are some examples and their possible data sources. 1. F  eedback on products or services – this is a vital piece of information that every business should care about. Customer feedback can be collected in different ways: a) Customer Surveys – Post purchase survey or feedback letters or emails are one of many ways to collect customer feedback. Companies like TrustPilot25 and Reevoo26 provide feedback collection services to other business. These companies also share customer feedback to empower other potential customers to make informed decisions based on past customers’ experiences. b) Social listening – customers talk about products, services and brands very openly on social media – businesses can listen to and participate in these conversations.There are many examples you will find on Twitter or Facebook where consumers or customers are giving negative feedback about various products, from antiviruses to mobile phones. Many brands use social listening as a proactive customer feedback tool to implement any changes/improvement needed. Domino’s Pizza is one such brand, and we will discuss this case study at a later point in this book. c) Verbal/hand-written feedback provided at store, by post or in emails. d)  Complaint submitted through website or emails. e)  Feedback shared at discussion forums. 2. P  oint of Sale (PoS) data – a retail point of sale system typically includes a cash register which comprises a computer, monitor, cash drawer, receipt printer, customer display and a barcode scanner and the majority of retail POS systems also include a debit/credit card reader.

Big Data revolution  25

Point of sale systems (PoS) collect the data about the product and their quality, frequency etc. from each customer and this can be used in analyzing and predicting customers’ buying behaviour.This data can also be used for customer profiling or can help businesses to build one-toone relationships with their customers. There are many cases where you can see how PoS data can add value to a business. For example, Target Stores in the USA use PoS data to predict expectant parents by analyzing their buying patterns based on their historic purchasing history.27 Tesco in the UK is also very wellknown for its use of PoS and loyalty card data to increase sales, improve processes and provide best service to their customers. 3. Store or website usage data – now technologies are helping businesses to identify customer/visitor activity on their website and in their physical stores. With the help of this data businesses can make changes to the offerings that best suit their customers’ needs. Business can work on the design and layout of the store and website based on users’ activity information collected. Google Analytics28 is one of the most popular services for e-retailers who want to track activities in their online stores. With the help of Google Analytics, businesses get a lot of insightful information such as new/old visitors, popular pages, popular sections of the website, popular products etc. This information helps businesses improve the look, feel, messaging and offering of their operations. That leads to satisfied customers and better sales conversion for the business, among many other things (see Figure 2.1).

FIGURE 2.1  A

typical data view of website visits

26  Big Data revolution

Similar to Google Analytics, Retail Next29 is one among few companies that have developed the technology to help businesses track and learn from their customers/visitors in their physical stores. Information collected can help businesses make many possible variations such as changes in the layout of the store, or changing the location of campaign posters etc. Employee data Data related to employees have been of immense value for a long time, and traditionally this was only available (or identified) from a few data sources. Now businesses can access different data from multiple sources that touch the different dimensions of an employee–employer relationship. 1. E  mployee Satisfaction Data – there are established ways of collecting employees’ satisfaction and feedback in many organizations. From surveys and verbal feedback to social media conversations, employee opinions can be collected to see how satisfied they are and what can be done to keep them happy. 2.  Employees’ activity/productivity data – businesses now have access to diverse means of gathering and analyzing data related to employees’ productivity. From company-owned mobile phones, emails to VPN network access, businesses will be able to collect and act on the data recorded about their employees. For example, sales teams’ daily updates, their car’s location and smartphone’s tracking data can easily tell you how efficiently they are spending their time with prospects. It is also possible to establish a pattern between converted sales versus your sales person’s predictions about the conversion of sales leads. This pattern (identifiable in the data itself) can then be related to the existing sales pipeline to predict future sales and booked revenue. In 2013 Yahoo’s then newly appointed CEO Marissa Mayer came into the limelight (and a bit of controversy) when she banned remote working for all Yahoo’s employees – she was highly criticized for this step, and we don’t want to comment on this decision, but we would like to draw your attention to the method by which she arrived at this decision. Like many technology companies, employees worked from home in Yahoo by logging into their personal and secured network from a remote location and when they were not working from a remote location, they were expected to work from office. Marissa recorded that there were relatively fewer cars parked on the company campus and at the same time a lot of employees were not logged in remotely to the system. On the basis of these two pieces of information she concluded that

Big Data revolution  27

many employees might not be working while they should be working from home so she decided to ban the remote working policy altogether to increase productivity – the point is, she had access to the data which helped her put together the whole picture. 3. Employees’ interest and innovation data – employees’ conversations on internal and external forums, social network and emails can also provide a lot of information about their areas of interest and expertise – this information can be utilized by business managers and leaders to identify innovators, thought leaders and influencers to help and promote a company’s brand. This data can easily be arranged within the organization with the help of the IT team. A relatively new concept of ‘employee advocates’ came into existence in around 2012 – the idea was for businesses to provide their employees with marketing messages and take advantage of their social network to spread the word about the brand. Businesses analyze the data from employees’ social media activities, identify the most influential ones in their employee network and use them to diffuse messages about the company in the wider social network. Employee advocates help their companies in making a stronger brand and in hiring talent for the company. Competition data Traditionally competitor analysis was performed based on market data available through government agencies (such as HMRC or Companies Hours), market research firms or internal research departments. Competitor news monitoring was performed manually for decades. Now brand monitoring is made possible electronically by crawling millions of webpages, social media and other portals and generating huge amounts of data automatically. With the help of technological innovation, now customers have also started to participate in generating data for brand and competition analysis – customers are expressing their views about products and services in the public domain on social networks, discussion forums and review portals, and their turnaround time is very quick. Also, as these conversations are happening in an independent environment, customers’ or users’ views about products or services are closer to reality.

Summary Like anywhere else in life, the right decisions at the right time can have a massive impact on the growth of businesses. To make sure there is no scope for error or ignorance, businesses try to make decisions backed up by data

28  Big Data revolution

and reason as far as possible. We have seen the impact and example of some smart data driven decisions and how data has contributed to the growth of a company. We have also looked back to see the trail of data and business connections – business intelligence and data-based decisions are not new but were not as popular among managers as they are now. One of the reasons for this is the invention of the modern technologies and access to meaningful and right data at the right time. In this chapter, before touching upon Big Data and its type/components, we have discussed data, analytics, statistics and computing. We have briefly covered essential data components such as warehousing, RDBMS, mining and BI. Then we discussed Big Data and its components – structured data and unstructured data – and how there has recently been a huge growth in unstructured data, before considering some examples where unstructured data is being used in some main business areas regarding customers, suppliers and employees.With the inception of Big Data technologies like Hadoop and Hive, we have come far, but there is still a lot to invent and improve in this space in the future. In the next chapters, we will be covering concepts of value creation in businesses and how data (and Big Data in particular) can be considered as a big advantage for any business in this age.

Notes   1  Monopoly Power – http://economicsonline.co.uk/Market_failures/Monopoly_ power.html  2  www.theguardian.com/media/2016/jan/20/waitrose-lidl-aldi-tv-ad   3            www.telegraph.co.uk/finance/newsbysector/retailandconsumer/12084091/ Waitrose-sales-suffer-Christmas-dip-while-John-Lewis-jumps.html  4  www.campaignlive.co.uk/article/1361396/waitrose-pick-own-offers-loyaltyscheme-helped-outpace-market#  5  www.campaignlive.co.uk/article/1363335/waitrose-hails-success-game-changing-pick-own-offers-expands-scheme  6  https://growthhackers.com/growth-studies/airbnb  7  w ww.businessinsider.com/uber-ceo-travis-kalanicks-success-story-20149?IR=T  8  http://expandedramblings.com/index.php/uber-statistics/   9  Bruce Henderson – www.bcgperspectives.com/classics/author/Bruce_Henderson/ 10  Michael Porter – www.hbs.edu/faculty/Pages/profile.aspx?facId=6532 11   Cyclopaedia of Commercial and Business Anecdotes – https://archive.org/ details/cyclopaediacomm00devegoog 12  http://thenextweb.com/in/2015/04/06/indian-fashion-retailer-myntra-is-closingits-site-on-may-1-to-focus-on-app-based-sales/ 13  www.telegraph.co.uk/finance/10967104/Kiddicare-sold-for-2m-to-Endlessprivate-equity.html 14  www.crunchbase.com/organization/hootsuite 15  www.-01.ibm.com/software/data/bigdata/what-is-big-data.html

Big Data revolution  29

16  www.academia.edu/12383046/Big_Data_Manipulation-A_new_concern_to_ the_ICT_world_A_massive_Survey_statistics_along_with_the_necessity 17  www.amazon.com/Scholar-Research-Librar y-Problem-Solution/dp/ B005ZN4N8C/ref=sr_1_1?s=books&ie=UTF8&qid=1339603976&sr=1-1& keywords=The+Scholar+and+the+Future+of+the+Research+Library 18  http://4840895.blogspot.co.uk/2009/04/history-of-dbms.html 19   www.bbc.co.uk/history/historic_figures/berners_lee_tim.shtml 20   www.oreilly.com/data/free/release-2-issue-11.csp 21  MapReduce: Simplified Data Processing on Large Clusters (Jeffrey Dean and Sanjay Ghemawat, 2004). 22  https://gigaom.com/2013/03/04/the-history-of-hadoop-from-4-nodes-tothe-future-of-data/ 23  http://swampland.time.com/2012/11/07/inside-the-secret-world-of-quantsand-data-crunchers-who-helped-obama-win/ 24   https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century 25   https://uk.trustpilot.com/ 26   www.reevoo.com/ 27  http://techland.time.com/2012/02/17/how-target-knew-a-high-school-girlwas-pregnant-before-her-parents/ 28   www.google.co.uk/analytics 29   http://retailnext.net/

3 CREATING VALUE FROM BIG DATA

The term ‘Value’ in general conversations and in a business context can mean many and different things. We must therefore decide and define the perspective we apply in this book before then determining how Big Data can affect value. The word value stems from the Latin word valere and can be defined broadly as the importance, worth or usefulness of something. In monetary terms it often means the material or monetary worth of something. In this book we take our perspective from an organizational view point and aim at practising managers in commercial organizations that seek to create value in monetary terms, and hence creating value ultimately has a monetary value.What is true not only in relation to Big Data but in many other aspects of business is that value cannot always directly be measured in monetary terms and that proxy measures are needed to make judgments of value. In this chapter we will discuss generic drivers for value creation using a Proforma Income Statement and Statement of Financial Position for a fictitious commercial enterprise. It is worth pointing out that this book is about Big Data and not about finance and accounting, so those of you that have a good understanding of accounting and finance can skim read this chapter.This chapter gives generic examples as a lead into value creation from Big Data whilst the chapters that follow discuss the technology and case studies from Big Data investments.

Value drivers in commercial organizations Over time, value is created if revenue exceeds the cost of generating the revenue. This is a fundamental of business and it is worthwhile going back to basics when considering how value is created. Revenue is the lifeblood of all business and we should pay attention to it. Easier said than done, however. The characteristics of revenue is that it is uncertain and much can happen in the marketplace that changes our expectations of future revenue. Cost on the other hand is often, but not always, relatively certain. It is also a factor that the business has a degree of control over costs. So we have uncertain revenue and relatively certain costs as a basis for our thinking about value creation.

Creating value from Big Data   31

The question to keep in mind is the extent to which Big Data affects the level of uncertainty in the revenue and how it affects costs. Included in the cost is not only the expense of generating the income but also the cost of the invested capital used to generate the income. Return on invested capital (ROIC) has two components: the amount of invested capital relative to the margin (income). Value is created if either margin goes up without an increase in invested capital or if the invested capital is reduced at a constant margin. Margin in turn has two components that drive value: one is revenue and the other is costs. Invested capital has two components: one is working capital used in the operations of the business such as product inventory or the timing difference between payments received from customers and payments made to suppliers; the other is fixed assets used to generate the income such as buildings and machinery used for the running of the business. Figure 3.1 shows the value drivers of value creation. Job #1 is therefore to understand which value driver an investment in Big Data analytics affects and if the additional revenue exceeds the cost of the investment in Big Data. Understanding how value is created requires an understanding of which value driver an activity such as Big Data affects. Does Big Data offer faster, better, cheaper products and services? Is it driving higher revenue without an increase in costs or the same revenue at lower cost? Alternatively does Big Data drive a reduction in working capital or fixed assets? There can of course be a mix of the basic value drivers but the point of fundamental importance is to understand which value drivers an investment in Big Data activities affects. It is also important to recognize that the drivers of value have different characteristics and, importantly, do not have the same degree of predictability and accuracy in the assumptions that underpin the estimates of value creation.You should keep the following in mind when evaluating an investment in Big Data (or any other investment). Table 3.1 discusses the level of

FIGURE 3.1 The

drivers of value creation

32  Creating value from Big Data TABLE 3.1  Level

of uncertainty in value capture

Value driver

Value creation hierarchy

Level of uncertainty in value capture

Revenue

Unpredictable and hard to quantify

Costs

Predictable and quantifiable

Working Capital

Predictable and quantifiable

Fixed Assets

Predictable but with long timescales

This is the most uncertain value driver as it makes assumptions on growth in number of customers or increases in prices at a given sales volume. Costs are generally internal and offer a higher degree of certainty than assumptions on increased revenue. Working capital is largely under the control of the organization and can therefore be predicted with a higher degree of certainty compared with revenue. Fixed asset values are generally known by the business. If the value is generated by a reduction (disposal) of fixed assets there is uncertainty of the re-sale value of the asset and the time it may take to dispose of the asset.

uncertainty for the value drivers. At the end of the chapter we will revert to this table and discuss the potential for Big Data investments to change or reduce the level of uncertainty in the investment decision. To help us understand the value drivers we have created a proforma income statement and balance sheet of a fictitious business. We will apply these to give examples of the value creation potential when applying Big Data in different ways in the business. The proforma income statement and balance sheet A generic income statement is a starting point for understanding where and how value is created. We will use this to illustrate how Big Data is driving value creation. In the examples below we show the proforma income statement and balance sheet in a medium sized business with $100m in sales and $10m net income. The cost of goods to produce the company’s products is $40m resulting in a gross margin of $60m or 60% in our example. We have chosen a smaller business with a relatively simple financial model for illustrative purposes rather than modelling some of the large Big Data generating

Creating value from Big Data   33

companies such as Amazon and Facebook, which are large complex businesses. We also believe the potential for value-creating investment by smaller firms is just as attractive as larger firms with the development and growth of firms that offer data analytics as a service. The size of investment in Big Data analytics varies but is generally high. For instance, the investments in Big Data analytics by one telecom company runs into millions of dollars per year – these are real numbers and it shows the level of investment in a Big Data project. Capital investment Hadoop investment Running costs IT/Data warehouse Internal 30 data analysts and staff

$15m $2m $3m $3m

These are substantial investments also for a major telecom company. This level of investment can be justified not only due to the size of the telecom company but also the significant level of data that is generated daily throughout the network and the large number of customers.There are industries that have similar characteristics, such as banks and insurance companies, logistics companies with fleets of trucks and lorries loaded with many individual packages for delivery, in fact any organization where the number of customers and product units sold is high. A particularly suitable type of organization is one that has a significant presence on the Internet. Companies such as Google and Facebook generate a huge amount of digital data which can be analyzed but also smaller organizations such as e-commerce companies generate enough data to benefit from Big Data analytics. In our example we assume that the investment is Big Data as a service with limited fixed asset investments compared with the example of the telecom company. The development of setting up the analytics system is often provided by a Big Data consulting company, therefore expensed in year 1 and the annual costs cover support and analysis services by the analytics provider. The income statement (Table 3.2) helps us to identify the revenue and cost drivers for value creation. But we need also to consider the capital required to run the business and how Big Data could drive value creation by affecting the capital used in the business such as working capital and fixed assets. For the calculation of value creation we use Net Operating Profit After Tax (NOPAT) as this is the income before we consider the returns to the providers of capital, i.e. interest to the bank on loans and the dividend that the shareholders receive for investing in the company. Our example balance sheet (Table 3.3) is relatively simple and shows the main components that make up assets and liabilities. The working capital is

34  Creating value from Big Data TABLE 3.2  Proforma

income statement (values in $ million)

Sales Revenue Cost of Goods Sold (COGS) labour & material Gross Margin (100-40) Overheads - General & admin (5) - Sales expenses (15) - Customer service (5) - R&D (10) Depreciation Operating Income - Taxes Net Operating Income After Taxes (NOPAT) - Interest expense Net Income - Retained earnings - Dividend TABLE 3.3  Proforma

100 (40) 60 (35)

(10) 15 (3) 12 (2) 10 (6) (4)

60%

15% 12% 10%

balance sheet

Assets Cash Accounts Receivable Inventories Net Property, Plant & Equip Total Assets

5 25

Working capital Working capital

20 100

Working capital Fixed assets

Goods invoiced but payment not yet received

150

Liabilities Accounts Payable

25

Working capital

Bank Debt Equity

50 75

Capital Capital

Total Liabilities

150

Supplies received but not yet paid for Share capital $50 and accumulated retained earnings $25

current assets minus current liabilities, i.e. assets and liabilities that can be converted into cash relatively quickly. In our example the working capital is:• •

 ash+Accounts Receivables+Inventories – Accounts Payable (5+25+20) C – (25) = $25m. The capital required to fund the business is bank debt and equity (50+75) to the tune of $125m.

Creating value from Big Data   35

In our proforma example we have estimated the cost of capital at 10% which includes both the cost of debt and equity capital. The value created is the calculation of return on invested capital (ROIC) from net operating profit after tax (NOPAT) divided by the invested capital (IC). ROIC = NOPAT / IC The invested capital (IC) is $125m sourced from equity, retained earnings and bank debt. NOPAT is $12m In our example the ROIC or value created is $12/$125 = 9.6%, in other words ROIC is less than the cost of capital of 10%. This business as it stands is profitable but does not create value. This means that any investment in Big Data activities must return more than 10% for it to be value creating and improve the current 9.6% ROIC.

From an investment perspective there is often a time lag between capital investments and the subsequent change in revenue and/or costs. This means we have to consider the time value of money. Money today has a higher value than money in the future.We will therefore introduce net present value (NPV) as a method for analyzing the flow of cash out of the business (investments and expenses) and cash into the business (revenue or reduction in invested capital) and use this tool to assess the value of the investment decision. A positive net present value indicates that the projected earnings generated by a project or investment (in present dollars) exceeds the anticipated costs (also in present dollars). Generally, an investment with a positive NPV will be a profitable one and one with a negative NPV will result in a net loss. In addition to projecting the cash cost and the cash income from the Big Data investment you need to establish the discount rate for the calculation of the net present value. In our example we have estimated the cost of capital to be 10%, however in all projects including Big Data there is a risk that the project will not deliver the projected outcomes. We therefore need to add a factor to compensate for the risks. In our example we will apply a discount rate of 15% which means we think the risk premium is 5% on top of the capital costs. In our examples we have used an investment in Big Data services in year 1 of $0.4m with annual running costs of $0.1m. Since Big Data is a fast developing technology we have assumed that the life of the investment is 5 years with no residual value in the investment or contribution to cash after 5 years.

36  Creating value from Big Data

However, not all things in a business can be directly measured in monetary terms. We therefore need to consider other means to measure value creation.

Market value and non-financial values Value is one of those terms that we hear and sometimes use ourselves on a regular basis. The meaning of the word value can have many meanings and we start by defining what we mean by value. Value in data and information can be defined in many different ways. •

•

• •

 alue to society – an open society and sharing of information, such V as how information was used to support the Arab spring, a series of anti-government protests, uprisings and rebellions that spread across the Middle East in early 2011. Value to the government – an increasing amount of digital data and analytics is used by governments to improve efficiency, provide information and communicate with its citizens. The UK tax department is a major user of Big Data, including social media, to investigate potential tax fraud. Value to you and yours – applications such as Facebook, WhatsApp and other social media offer constant and real-time communication with family and friends. Financial value to commercial organizations.

This book is aimed at understanding how value is created in commercial organizations and we start by understanding value at the highest level. In the case of a commercial organization this is the enterprise value of the organization. Traditional methods of valuing a company multiply the value of a share and the numbers of shares plus the value of the debt – this is called the enterprise value. In a publicly listed company shares are traded on a stock exchange and the market value is easy to calculate. In private companies the share value is more difficult to calculate when there is no public market for the share. Another way to value a company is to use the book value of the company as this reflects the capital investments that has been made over time. Comparing calculations of book value and market value over time raise an interesting question of how to value companies and what drives this valuation. Research by Accenture shows that book value has reduced from 80% of market value to 25% over a 20-year period. What it means is that investments in tangible assets recorded in the balance sheet of a company have less impact on the value drivers today than in the past. There are several potential explanations for this but one is that non-financial factors have a higher impact on market value today and traditional accounting measures are not able to capture these

Creating value from Big Data   37

in the drivers of value. These non-financial factors are often referred to as Economic, Social and Governance (ESG) factors. Research into ESG factors suggest there are links between these and the enterprise value of a company. The link to Big Data is how Big Data can drive value creation in the ESG factors. Core non-financial values in the ESG framework are customer relations, society, innovation, environment, human capital and corporate governance. Breaking these factors further begins to help us identify where Big Data analytics can be translated into non-financial value drivers and hence value creation (see Table 3.4). The ESG provides a framework for understanding how Big Data can create value and then link this to the Income Statement and Balance sheet of the company. It is relatively easy to estimate the investment and costs in Big Data analytics and activities. What is more difficult is to estimate the potential revenue.We therefore need to identify proxies for value drivers and make judgments of the financial value of these proxies. We have included some of the relationships where investments in Big Data have an impact whilst acknowledging that there are also other important relationships. Big Data is a natural consequence of digitization of sales and product flows in the supply chain.When the Internet emerged as a major shift in how people and companies communicate, the projections for growth in e-commerce were exponential.These early projections have been largely correct, albeit with longer timescales than originally projected.As is often the case, it is the convergence between what technology can offer, investments in infrastructure and the acceptance by and changing behaviour of customers. Today e-commerce and online retailing is a major force behind the growth and need for Big Data. Britain is leading the pack with 17% of retail sales online followed by the US at 13%, Germany at 12% and France and Sweden at 9% of retail sales.1 It was estimated that 1.2bn packages will be sent by retailers in the UK in 2016, delivered from warehouses covering 40m square meters. Delievries by lorries have increased 4% from 2015 to 2016 whereas overall traffic has only increased by 1%. This shift in how we buy goods has fuelled the development of Big Data analytics since much of the information has been in digital form from the beginning. A retailer such as Tesco has 3,500 stores in the UK, with 40,000 products generating 100,000 data points to be tracked. Another consequence of the growth in e-commerce is the physical distribution of goods with thousands of lorries and delivery trucks criss-crossing the country.The transport industry estimates there is a shortage of 45,000 drivers to drive the lorries. Maybe driverless lorries is the answer, requiring even more data capacity to keep track of them as the travel around the country. Companies such as FedEX and DHL are as much data centres as they are trucking companies.

TABLE 3.4  Non-financial

drivers of value creation

Core nonfinancial drivers

Customer relations

Society

Innovation

Environment

Human capital

Corporate governance

Key metrics

Customer satisfaction

Product and service developments

Environmental impact

Employee engagement

Ethical integrity

ESG factors

Customer retention; Customer loyalty; Trust; Reputation; Price, product, service quality

Public perception; Supply chain management Community investments; Stakeholder dialogue; Media coverage

New products and services; R&D expenditure; Value of patents; No. of new products

Energy efficiency; Waste reduction; Recycling; Reduction in pollution from transport

Health and safety; Recruitment & retention; Training; Performance management; Equality & diversity

Reporting & transparency; Ethical code of conduct; Board composition; Equality & diversity

Creating value from Big Data   39

Investments where Big Data can create value For commercial organizations the most value creating decision a company can take is to increase price. According to research of 2,348 companies by McKinsey, a management consultancy, a 1% increase in the following activities, all else being equal, results in an increase in profit by: • • • •

1% increase in price – 11% increase in profit 1% reduction in variable cost – 7.8% increase in profit 1% volume increase – 3.3% increase in profit 1% cut in fixed costs – 2.3% increase in profit

It often comes as a surprise to many business people that the highest value is created by customers being willing to pay more for a product or service rather than a higher volume of customers paying the same price. Reducing variable costs, i.e. the cost of producing the product, is fairly attractive, however cutting fixed cost is the least profitable action that a business could take. Investments in Big Data are often focused on the customer, either on an increase in number of customers and sales revenue or an increase in the revenue per customer. What this amounts to is a more fine-grained segmentation of the customer base. Telecom companies have large numbers of customers and very frequent interactions with the customer when they browse the internet on their smart phones and uniquely, the telecom provider also knows the location of the customer.The potential for value creation is therefore significant. The current industry standard is to use 6–7 segments such as high data users, international travellers and early tech adopters etc. The key financial metrics in the telecom industry are Subscriber Acquisition Costs (SAC), Churn (percentage of customers that don’t renew their contracts) and Average Revenue Per User (ARPU).The obvious value drivers are to reduce SAC, reduce Churn and increase ARPU. The industry is therefore investing heavily in Big Data analytics to better understand their customer base, make profitable decisions about which customers to spend marketing costs on, which customers are costly to serve and how to encourage increased usage. The nature of Big Data investment addressing customers and revenue is significantly more uncertain than a cost saving. The outcome is entirely dependent on the accuracy of the assumptions that underpin the decision. Another challenge to consider is the availability of accurate data for establishing the base line and compare this with other possible investment alternatives – this is A/B testing. In general, if the business has a poor handle on its customer data then the investment decision is a shot in the dark.This then becomes a chicken and egg situation where investment in improved accuracy through investment in Big Data is a prerequisite for information and data that support the investment in Big Data.

40  Creating value from Big Data

There are some proxies that can be applied to get a better handle on the assumptions for establishing the value creation potential. In the next section we will provide examples of how an investment in Big Data could create value. We will start with identifying customer segments that are willing to pay higher prices and increase the average spend per customer and we will use the telecom industry as examples of how to establish some of the value creation related to customers. Creating value from additional revenue from existing customers – higher prices This is an important aspect of value creation. In many businesses the cost of customer acquisition is high, hence additional revenue from existing customers can be highly profitable. This is particularly the case when the gross margin is high (i.e. the cost of producing the product or service is low). In the telecom industry an extra customer costs very little to service as the network is already built and as long there is no need to add capacity because of customer growth the gross margin is near 100%. Using Big Data the game is to identify customer segments that are less sensitive to price such as early adopters. By cross-selling or up-selling to existing customers the revenue per customer could be increased further. High gross margin markets are typical for digital businesses such as Facebook and Google where the economic model is one of increasing returns to growth and size and growing market share is highly profitable. Returning to our proforma example the additional revenue from an increase in price of 1% for 20% of existing customers results in an average revenue of $102.This would add $2m to the revenue and $1.2m of additional cash at 60% gross margin. Assuming $0.1m in additional marketing and selling cost. The resulting NPV on the investment is $3.0m. Table 3.5 shows the power of increasing price. This is by far the strongest value creating driver. Getting higher prices from existing customers is highly attractive and for telecom companies increasing the ARPU (Average Revenue Per Customer) is the key focus for the marketing department. TABLE 3.5  Creating

Year 1 Year 2 Year 3 Year 4 Year 5

additional value from existing customers

Investment

Additional marketing spend

Ops costs

Increase in cash flow from increased revenue

Net cash flow

$0.4m

$0.1m $0.1m $0.1m $0.1m $0.1m

$0.1m $0.1m $0.1m $0.1m $0.1m

$1.2m $1.2m $1.2m $1.2m $1.2m

$0.6m $1.0m $1.0m $1.0m $1.0m

NPV

$3.0m

Creating value from Big Data   41

By identifying segments with high propensity to spend, such as early adopters, investments in Big Data analytics can prove very attractive. Creating value from customer satisfaction Customers can impact value creation either through revenue growth from selling more of the product or service, or a reduction in costs to deliver the product or service. Let’s start with the reduction in the cost of producing, selling and servicing the products. In most businesses the customer has more than one interaction with the products and services. Increased competition makes differentiating products increasingly difficult hence in many sectors it is the service element of the customer journey that creates both costs and opportunities to retain and possibly enhance the relationship with customers. In the telecom industry analyzing the operational performance of the network is critical for offering a high service level. The amount of data and information to manage the networks are very large and Big Data analytics is used to combine real time performance data with forecasts for future events. For example, a big football event with thousands of fans filling the street demands good service from the network. Operationally a telecom operator is able to identify the presence of high value customers at the football game and provide a priority service. This has to be done automatically and only predictive algorithms and the fact that each customer can be identified makes this possible. The question is how the cost to serve can affect the overhead element in the proforma income statement. In our example in Table 3.6 we estimate that the Big Data investment would allow the customer service department to service 10% more customers per day and that the cost to service customers is $5 million per year primarily in staff cost (see proforma income statement). If the improved productivity requires 10% less staff the annual saving is $0.5 million in staff costs. However, the restructuring of the service department will cost $0.3 million in year 1. It seems that in this example the increased efficiency is costing the business approximately $0.3 million in year 1 (saving $0.5m – investment $0.4m – restructuring $0.3 – Big Data operational cost £0.1m) and then saving $0.4 the following years by investing in TABLE 3.6  Creating

Year 1 Year 2 Year 3 Year 4 Year 5

value from customer satisfaction

Investment

Restructuring

Ops costs

Saving

Net cash flow

NPV

$0.4m

$0.3m

$0.1m $0.1m $0.1m $0.1m $0.1m

$0.5m $0.5m $0.5m $0.5m $0.5m

−$0.3m   $0.4m   $0.4m   $0.4m   $0.4m

$0.73m

42  Creating value from Big Data

Big Data. A break-even calculation suggests it would take 1.6 years to repay the investment. A net present value calculation using the 15% discount rate results in an NPV of $0.73m. This example is a sound investment as the investment costs and the savings are fairly accurate and predictable. This is an investment that should go ahead. Creating value from reduced subscriber acquisition costs One way to establish the subscriber acquisition cost (SAC) is to add up costs related to sales, such as shops and other distribution channels, costs related to marketing and promotions, discount schemes and other activities directed at winning a customer. Divide this by the number of customers and you have a proxy for the customer acquisition costs. It is generally known in the marketing industry that a significant portion of marketing spend is wasted but no one can tell which activities are a waste of money. The problem we are trying to solve is how to target customers better and reduce the average costs of acquiring a new customer. Investments in Big Data can then be measured against either the reduction in SAC or against an assumed increase in customers multiplied by the revenue per customer. A reduction in SAC can be through better targeting of customers or more fine-grained segmentation. Applying a similar investment as described in the proforma case of serving 10% more customers could look like the following.The sales cost for the business in our proforma example is $5m of which the direct marketing cost is $1m per year. With 1 million customers making purchases the cost of marketing is $1 per customer. It is nearly impossible to build enough understanding of the customer segmentation through traditional income and demographic segmentation methods. Using large data sets from a range of sources such as purchasing patterns, social media activities, etc.,2 Big Data provides analytics that can improve the accuracy of targeting marketing of the customer base. In our example the cost of marketing per customer is $1 per year from an average revenue of $100 per customer. Using email push marketing the 1% marketing cost does not seem high enough to warrant major investments. However, since the customer churn in our example is nearly 50% the company must replace half its customer base each year, i.e. find 500,000 new customers each year. The question is, can a $0.4m investment in Big Data analytics and a refined customer segmentation and better targeting of marketing messages create value? What value could be created with better segmentation and targeting of those customer segments that seem to have a higher propensity to purchase than others? Which segments of the potential customer base are more likely to become repeat customers rather than one-off buyers? In our example in Table 3.7 a better understanding of the customer base and more targeted direct marketing could result in a reduction in churn

Creating value from Big Data   43 TABLE 3.7  Creating

Year 1 Year 2 Year 3 Year 4 Year 5

value from reduced subscriber acquisition costs

Investment

Ops costs

Saving from reduced marketing spend

Net cash flow

NPV

$0.4m

$0.1m $0.1m $0.1m $0.1m $0.1m

$0.2m $0.2m $0.2m $0.2m $0.2m

−$0.3m  $0.1m  $0.1m  $0.1m  $0.1m

$0.01m

from say 50% to 30% resulting in 200,000 customers becoming repeat buyers. The direct marketing spend could therefore be reduced by $0.2m (200,000 customers x $1 marketing spend per customer). The value created would be a net investment of £0.3m in year 1 and a saving of $0.1m for the next 4 years. Assuming this saving would continue during the lifetime of the investment, the net present value is calculated at an NPV of $0.1 million. This is an investment that should not go ahead as the NPV is virtually nil. As Table 3.7 shows, it is not always a given that investing with the aim of reducing cost will pay off. However, customer acquisition cost is often high so let’s investigate keeping the marketing spend and targeting customers more effectively. Creating value from more effective marketing – acquiring more customers An alternative to reducing the marketing spend through reduced customer acquisition cost is to maintain the marketing spend and acquire more customers. Growing the customer base is the ambition of most organizations and in general marketing is the way to achieve this. We continue using our proforma example (see Table 3.8) with the marketing spend continuing at $1m, and with a more effective targeting of marketing this could increase the number of customers by say 10,000 and produce an increase in revenue of $1m given the $100 average spend per customer. With a gross margin of 60% for an average customer revenue of $100 it would result in an increase in additional cash of $0.6m. The NPV of the $0.4m investment over 5 years calculates to an NPV of $1.33m. By maintaining marketing spend, and with more effective and targeted marketing, an investment in Big Data analytics could yield an NPV of $1.33m, so this is a sound investment. This serves to show that revenue is often a stronger value driver than cost reduction, however, a reduction in costs is less uncertain than predicting higher revenue.

44  Creating value from Big Data TABLE 3.8  Creating

Year 1 Year 2 Year 3 Year 4 Year 5

value from more effective marketing

Investment

Ops costs

Increase in cash flow from increased revenue

Net cash flow

$0.4m

$0.1m $0.1m $0.1m $0.1m $0.1m

$0.6m $0.6m $0.6m $0.6m $0.6m

$0.1m $0.5m $0.5m $0.5m $0.5m

NPV

$1.33m

Creating value from customer retention value Customer retention is often related to how the customer perceives the value of the product and service. When the product or service is of high value to the customer and it is not costless for the customer to change to another supplier, customer retention is often high.The key is to create switching costs such that it is inconvenient or ‘costly’ for the customer to buy the product or service from a competitor. For example, the modern farmer is a high tech business nowadays, with GPS controlled tractors and harvesters that work out the most cost-effective route when harvesting. In Big Data terms this can equate to the information value for the customer such as providing the farmer with a combination of market prices for the crop, weather predictions and market trends. This can be of high value to the farmer. In the case of potatoes, the application data and agronomy can drive yield productivity up from 30% to 50%, depending on type of soil, which is a significant value creation for the farmer.3 Another area where optimization is beneficial for the farmer but also to society is the reduced use of chemicals and pesticides. There are 2.2 million farms in the US and the average farm spends $111,000 on chemicals; any reduction would not only improve profitability but also make an important contribution to the environment. The opportunity to computerize farming and optimize the use of chemicals and pesticides pits two different suppliers against each other in a data war. For example, one is the agrichemical business Monsanto and the other the farm equipment maker John Deere. Monsanto realized that data could increase world-wide crop production by 30% through data-driven planting advice – and deliver an estimated $20bn economic impact. However, collecting the data and then planting and harvesting needs sophisticated and data-connected farming equipment. This needs sensing equipment that measures, for example, salinity and other soil characteristics, and a Global Positioning System (GPS) and Geographic Information System (GIS) that tracks the environment for changes. The farm equipment manufacturer John

Creating value from Big Data   45

Deere is a provider of systems that collect, analyze and interpret the data provided by the sensors in the farm equipment and data from other sources. Once John Deere has historical trend data and information about the specific farm it would take time for the farmer to build up this data base if switching to another supplier of farm equipment. A point of contention in the development of Big Data is who owns the data.4 Is it John Deere or is it the farmer? Depending on the answer to this question the method of calculating the value is different. Seen from John Deere’s perspective, the value creation is similar to the examples above. For the farmer, the value creation is the value of the customer surplus, i.e. the value of the information for additional profit less any costs associated with collecting the data. Supply chain management So far we have looked downstream of the company’s activities, mainly focused on the customer. For many organizations managing the supply chain provides competitive advantage by reducing costs through efficiency, ensuring products are available at the right place and at the right time. Effective and efficient supply chain management is a critical part of business performance. Supply chains are becoming longer and more complex, hence data, data flows and decision processes affect the value creation potential in a major way. The prime value creation in supply chain management is avoiding loss of sales due to stock-out and carrying slow moving stock in inventory.The 800lb gorilla in inventory and supply chain management is the world’s largest retailer, namely Walmart. Walmart runs a 40-petabyte database of sales transactions for the previous week. With both the large volume of transactions and a very dynamic business environment, with 2-for-1 offers, discount schemes and the competition constantly changing their prices, it would be impossible to manage the revenue and margin across all Walmart stores without Big Data analytics. In addition to sales data, Walmart collects data streams from another 200 sources such as meteorological data (ice cream sales go up when there is a heat wave), economic data (how much money is there in the pocket of consumers), social media data (who is saying what about Walmart) and gasoline prices (higher gas prices and people will switch to smaller stores nearer to home). The interesting point from an analytics perspective is the challenge of combining fast changing weekly sales data and slower trending data such as weather, economics and gasoline price changes. The supply chain can also be used to create scarcity. Zara is the brand name of the Spanish clothing manufacturer Inditex. Inditex was founded in 1972 by Amancio Ortega and has a different configuration of its supply chain compared with its main rival H&M. H&M designs and orders its garments

46  Creating value from Big Data

from manufacturers in the Indian sub-continent, often 6-9 months ahead of the season.This requires excellent buying and planning but if the buyers have over-estimated the sales of some items significant price reductions are needed to sell out the range before the next season’s garments arrive in the store. Zara however operates very differently. When Mr Ortega started his retail business he sourced all his garments from small, sometimes single individuals in Spain that stitched and made the garments to Mr Ortega’s specification.This meant that as the business grew the team at Zara had to learn how to manage thousands of suppliers in an effective way. Today Inditex is vertically integrated with most of the production taking place in Spain. The short distance and close collaboration offers a very different way of applying data in the value chain. Zara has 7,000 stores in 91 countries and yet it can design, produce and deliver a new range of clothes in 4–6 weeks. Zara’s designers create approximately 40,000 new designs, of which 10,000 are selected for production, creating more than 300,000 new stock-keeping units (SKUs). This can only be managed through data and more importantly sharing of data between designers, production units and supply chain managers. Electronic point of sale (EPOS) data from the stores is immediately available to all parts of the company. This means that decisions can be taken fast based on real time data of what is selling and where.The value creation from the ability to handle the large amount of data is in the working capital section of the balance sheet. Zara is able to significantly reduce the amount of garments that sits in the supply chain and in the stores and not being sold. The industry average of unsold items is 17 to 20% whilst in the case of Zara it is less than 10%. This also means less discounting is required to clear the stock for the new season’s garments hence also increases the gross margin leading to higher profitability. It is sometimes argued that Zara is better at predicting the fickle changes in fashion but it is actually its ability to react to changes in fashion throughout its operation that is the key success factor. All this is based on managing the flow of data and information in real time. According to research by Deloitte5 the main developments of Big Data and advanced analytics in supply chains are being integrated into (1) optimization tools, (2) demand forecasting, (3) integrated business planning and (4) collaboration & risk analysis. The report identifies that competence in analytics was deemed to be the fifth most important competence today whilst it is expected to be the most important competence for supply chain performance in the future. Research by Accenture has found that there are three key practices that enhance the ROI of Big Data analytics6 in their supply chain strategy. First, the ROI benefits greatly from taking Big Data investment decisions with an Enterprise-wide perspective; second, it clearly defines the issues and opportunities that an investment in Big Data analytics is attempting to address, often supported with smaller pilot projects to confirm initial hypothesis; and third, it operationalizes Big Data analytics in the

Creating value from Big Data   47

day-to-day activities rather than as a separate function within the business. Some key data from the Accenture report suggest: Results from Big Data investments: • • •

Improvements in customer service and demand fulfilment of 10% or greater (46% of respondents) Faster and more effective reaction time to supply chain issues (41% of respondents) Increase in supply chain efficiency of 10% or greater (36% of respondents)

Operationalization of Big Data into the day-to-day operation is reported to significantly increase shortening of the order-to-delivery cycle and increase supply chain efficiency according to Accenture. Product-based businesses often have both long and complex supply chains for input in the manufacture of products. Over the past 30 years much component manufacture has moved from near the user markets in the US and Western Europe to Asia and China. Simple components were increasingly easy and inexpensive to ship from manufacturing plants in Asia to assembly lines in the US and Europe. In the past 15 years more products have been manufactured and assembled in Asia, resulting in more of the value-added taking place in the supply chain rather than with the final product owner.This has increased manufacturing skills and knowledge in Asia and in the case of Apple, the iPhone is made of components from a large number of specialist suppliers of electronics components in Asia, assembled as a finished product by Foxconn in China and shipped to customers around the world. The development of high technology manufacturing companies in Asia has also created new and competitive companies that start to compete with the traditional Western companies in the telecom supplier market (e.g. Huawei from China), cars (e.g. Hyundai from South Korea), phone handsets (Samsung) and many others. This increase in competition results in product competition but also in the efficiency of the supply chain where the speed of information flow, and coordination of suppliers and shippers becomes the focus. Well-managed supply chains have the potential to reduce inventories and obsolescence that drives value through reduced working capital. The value creation potential of lower costs of goods in the P&L can be attractive. These benefits are set against the costs and risks in complex and difficult to manage supply chains. Returning to our proforma example (see Table 3.9), a reduction in the cost of goods by 2.5% of the 40% in COGS to 39%, all else being equal, would result in an increase in cash flow by $1m from an investment of $0.4m in year 1 and an NPV of $2.67m. In fast moving consumer goods (FMCG) markets the difference between good profits and poor results is how accurately demand can be predicted and where the supply of products should be directed to minimize stock-out and non-delivery to the end customer.

48  Creating value from Big Data TABLE 3.9  Creating

Year 1 Year 2 Year 3 Year 4 Year 5

value from effective supply chain management

Investment

Ops costs

Increase in cash flow from reduced COGS

Net cash flow

$0.4m

$0.1m $0.1m $0.1m $0.1m $0.1m

$1.0m $1.0m $1.0m $1.0m $1.0m

$0.5m $0.9m $0.9m $0.9m $0.9m

NPV

$2.67m

A major household name in toothpaste is using Big Data to ensure the product is always available on the shelves. In this example the success is not only the analytics capability of the toothpaste manufacturer but also how other providers of data are involved in the chain of decisions. Figure 3.2 describes the principles behind the information flow. The key data source is the EPOS data from retailers scattered around the country where toothpaste is sold. The toothpaste manufacturer has the option of buying the EPOS data directly from the retailers and aggregating the data or buying the data from EPOS aggregators such as Nielsen or Kantar Worldpanel. Given the cost of buying and aggregating the data it is often more cost effective to buy the data from an aggregator as this will also provide information on sales and pricing of the competing toothpaste brands. Of course the competing brands of toothpaste are likely to buy the same data, hence any advantage is only going to come from being better at analyzing the data and taking decisions about pricing, marketing campaigns and supply chain management. In fact, similar to Zara, it is the ability to share data with every part of the supply chain and take decisions that matter, and to take decisions faster than the competition. The challenge and opportunity is to have a real-time and an integrated view of pricing, volumes and product flow in the supply chain. The amount of data and the ability to take fast decisions is one of the main challenges in Big Data investments. This is where visualization software such as Tableau supports weekly or even daily decisions which affect the profits and hence value creation for the toothpaste manufacturer. However, do not underestimate the challenge in deciding what information to visualize and how. Those decisions alone will alter which decisions are taken and the results. In general, visuals such as graphs and pie-charts tend to work better than tables with numbers when several people are making decisions together. If a business is better at forecasting demand and replenishing the product on the shelves, this results in an increase in revenue of 2% or $2m in our proforma example (see Table 3.10). The improvement of $1.2m in cash flow

Creating value from Big Data   49

The Data Flow in Toothpaste DATA AGGREGATION ACNielsen KANTAR WORLDPANEL

EPOS

BUYS DATA FROM RETAIL

Manufacturer A

MANAGE Manufacturing

Manufacturer B Manufacturer C 1. Buys competor data from Aggregator

Pricing & Incenves Markeng Sales force planning

2. Data Analysis 3. Business Decisions

FIGURE 3.2 The

data flow in toothpaste

TABLE 3.10  Creating

Year 1 Year 2 Year 3 Year 4 Year 5

value from improved forecasting

Investment

Ops costs

Increase in cash flow from increased revenue

Net cash flow

NPV

$0.4m

$0.1m $0.1m $0.1m $0.1m $0.1m

$1.2m $1.2m $1.2m $1.2m $1.2m

$0.7m $1.1m $1.1m $1.1m $1.1m

$3.34m

results in a gross margin of 60%. This $0.6m increase in cash flow represents an NPV of $3.34m. So far we have discussed examples of calculations of value creation based on a set of assumptions. The next section will discuss innovation as a driver for value creation but also that fast and good decision taking is needed to capture the value that Big Data is creating. When data itself is the source of value creation So far we have looked at how data has provided information to take valuecreating decisions. However, there are highly successful businesses where the value creation is in the data itself. In the example on the toothpaste manufacturer we mention AC Nielsen and their role as a provider of data. In fact AC Nielsen and a range of businesses such as IMS Health all address very different markets with the common denominator of being providers of data. It is

50  Creating value from Big Data

interesting to note that all these businesses were founded nearly 100 years ago which serves to show that data has always been an asset of value. AC Nielsen was founded in 1923 in Chicago by Arthur C. Nielsen and the business was to give marketers reliable and objective information on the effectiveness of marketing campaigns. This is very much its purpose today but rather than asking customers how they perceived the campaigns almost all the data is collected electronically through a range of sources, analyzed, package and sold to clients that then use the data for taking decisions. IMS Health creates value for itself and its customers by having one of the most comprehensive sources of data in the health industry. The value is the creation of a network of sources such as drugs manufacturers, pharmacies, retailers, hospitals and any other organizations that buy and use drugs and consumables in the health industry. IMS Health has more than 29,000 sources of data which are then collated, analyzed and offered in over 10,000 different reports sold to clients all over the world. Innovation – data driven decisions using Big Data There has been an intense discussion for some time among economists and computer scientists on the extent or otherwise to which investment in IT and lately Big Data analytics drive profitability. Erik Brynjolfsson from MIT coined the term ‘the productivity paradox’ in 1993 where IT investments were increasing in the 1980s but there was no measurable impact on growth in GDP and other measures. This is corroborated by Robert Gordon in his book The Rise and Fall of American Growth (2016) which suggests that productivity growth in America has been flat since the 1970s despite significant investment in IT technology. Brynolfsson’s latest research7 on data driven decision-making (DDD) finds evidence that in fact DDD is associated with a 5%–6% increase in output and productivity, beyond what can be explained by traditional inputs and IT usage. Brynjolfsson’s argument is that it is not the investment in IT that matters but applying data for decision-making. Nesta in the UK surveyed8 500 UK companies and asked how they make decisions to grow their sales. In only 18% of cases did the respondents use data and analysis for decision-making. Forty-three per cent of the respondents in the NESTA research mostly or exclusively used experience and intuition in decision making. According to the research, the companies that used data are: • • •

twice as likely to run controlled experiments to see what works three times as likely to use customer data to develop their business strategy 25 per cent more likely to say they launch products and services before competitors

Creating value from Big Data   51

The research identifies ‘datavores’ as organizations that actively use data in taking decisions. The NESTA analysis show that many companies use trends and business intelligence (BI) reports. The biggest gap is how the data is displayed, where 84% of datavores use visualization and dashboards whilst only 49% of other companies use these techniques. The data also show that there is still relatively low use of controlled trials and statistical analysis such as regression analysis for both categories, hence there is scope for companies to steal a march on competitors by being more sophisticated in the use of data in taking decisions. This finding in conjunction with Brynjolfsson’s research on DDD suggests there are significant opportunities for companies that apply data driven decision-making such as Big Data analytics, who stand a good chance of outcompeting the competition. The NESTA research suggest that 46.4% of DDD companies report online customers impacting significantly on performance compared with 12.0% for companies with limited data in their decision-making. The point is, business success is a game of relative competitive advantage and those with better data, but importantly better decision processes, will win. Research by Forrester, a research firm, provides insights into how retailers grapple with the need for data but also the challenge of handling large volumes of data. ‘Retailers are overwhelmed by data’, say Dunhummby.9 In the research, 28% of retailers say that the inability to use the data that is available is a barrier to achieving customer centricity, with 66% saying that the main problem is achieving a single customer view. The mix of brick-and-mortar stores and online sales, often referred to as omnichannels, is a major headache for retailers, and 29% of retailers in the Forrester research report that conflict between the different channel owners hampers progress and reduces the potential value created. What these comments alert us to is that collaboration and joined-up thinking is as much the driver for value creation when the amount of information is large and complex. Although not the purpose of this book, it is clear that leadership and management decisions will have significant impact on the results from Big Data investments. Crowdsourcing and LEGO bricks – many of us have grown up with LEGO in our early years of exploration and have let our fantasy blossom through building with LEGO. What is special is not only the longevity of such a simple product as the LEGO brick, but how the company has harnessed Big Data to engage their user community online as a major source of ideas and input into the new product development process. Today LEGO has 150 known user groups and 100,000 active adult fans who regularly contribute ideas and feedback on new designs and products. If a design by the user community gets 10,000 votes and if the design subsequently is commercialized the innovator receives 1% of the total net sales for their product design.

52  Creating value from Big Data

In 2011, the Minecraft project received 10,000 votes worldwide within 48 hours and it has received 30,000 likes on Facebook pages. Over the years adult fans have uploaded more than 300,000 of their own LEGO creations and posted more than 4.5 million photos, drawings and instructions online.10 The size of the community would be very difficult to monitor and keep engaged without Big Data analytics such as sentiment analysis and unstructured data analysis of the thousands of conversations that continuously take place on the Internet. The obvious value creation is the sales revenue that is generated from new designs that already have at least 10,000 potential customers for a new product design. It also multiplies the product development resource from a couple of hundred LEGO employees to tens of thousands of LEGO enthusiasts around the world. A major cost to society is health care and looking after the elderly. Research by McKinsey,11 a consulting firm, suggests that if US health care were to use Big Data to drive efficiency and quality the sector could create $300–$450bn of value. McKinsey suggests the key value drivers using Big Data analytics estimate that the value created could be: • • • • •

$70–100bn Targeted disease prevention and data-enabled adherence programmes $90–100bn Alignment of proven pathways and coordination across providers $50–70bn Right care setting and reducing re-admission rates $50–100bn Payment innovation and performance transparency $40–70bn Accelerating discovery in R&D and improving trial operations

It is noticeable that about $160–200bn is related to the way new treatments and drugs are developed and applied in an effective and efficient way whilst the rest of the value created ($140–240bn) relates to co-operation between actors in the health system of which the ability to share data seems to be a major obstacle. The report suggests that the technological advances in handling large volumes of data – especially unstructured data from patient records – analytics and visualization that Big Data offers, provide opportunities for progress that was not available 20 years ago. The value created in the health care system can be applied in two ways. By reducing expenditure, government taxation could be reduced or alternatively more treatment of illness could be afforded. In Europe, McKinsey estimate that €100 billion in operational efficiencies could be achieved by governments. To enable this potential value creation, there needs to be innovation in not only the measurement and availability of patient data but also in how society can handle the massive volume of information that needs to be shared

Creating value from Big Data   53

between providers of technology, the health care system and government agencies involved in decision and allocation of money and resources. Big Data is having major impact on society through the way information can be collected both efficiently and in large volumes. This affects us as private citizens in our interaction with government agencies such as tax authorities and social services. In a conversation between a senior tax inspector in Denmark and one of the authors the tax inspector said that Big Data analytics has had a major impact on tracking down tax avoidance through not only official data but increasingly also through unstructured data sourced from social networks, not to mention the activities of the intelligence services where cyber space is becoming the new battleground for national security. In summary, we have shown a number of different ways in which value can be created. The important point is to focus on understanding the underlying drivers of value creation when we consider an investment in Big Data. A second point of importance is to understand the level of uncertainty in each value driver such that revenue is the most uncertain of the four value drivers. What this means in practice is to be explicit about the assumptions that underpin any calculations in deciding whether to invest or not in Big Data analytics. It may be useful to repeat the table shown in the introduction to this chapter (Table 3.11). TABLE 3.11  Level

of uncertainty in value capture

Value driver

Value creation hierarchy

Level of uncertainty in value capture

Revenue

Unpredictable and hard to quantify

Costs

Predictable and quantifiable

Working Capital

Predictable and quantifiable

Fixed Assets

Predictable but with long timescales

This is the most uncertain value driver as it makes assumptions on growth in number of customers or increases in prices at a given sales volume. Costs are generally internal and offer a higher degree of certainty than assumptions on increased revenue. Working capital is largely under the control of the organization and can therefore be predicted with a higher degree of certainty compared with revenue. Fixed asset values are generally known by the business. If the value is generated by a reduction (disposal) of fixed assets there is uncertainty of the re-sale value of the asset and the time it may take to dispose of the asset.

54  Creating value from Big Data

N.B. It is important at this point to remember that these are examples for illustrative purposes only and that the opportunities for value creation through Big Data are large and varied.

Summary In this chapter, we have discussed value creation and how investment decisions in Big Data can create competitive advantage in today’s data-rich business environment. The origin of value as a concept in business goes back centuries, but the meaning and importance are still the same. We discussed financial and non-financial value creation and we have presented examples of how value creation can be achieved through the use of Big Data analytics. Our method for calculating the value creating potential is to identify one of the four value drivers in the financials of a business and use the net present value of a Big Data investment to make decisions.

Notes   1  ‘All that is solid melts into air’, The Economist, 25 November, 2016.   2  Some of the data is internal but much of the data is purchased from other sources.   3 Thomas and McSharry (2015) Big Data Revolution, Wiley.  4 www.forbes.com/sites/emc/2014/07/08/who-owns-farmers-big-data/print/  5 Deloitte report. www2.deloitte.com/content/dam/Deloitte/global/Documents/ Process-and-Operations/gx-operations-supply-chain-­t alent-of-the-future042815.pdf   6 Accenture report. www.accenture.com/t20160106T194441__w__/fi-en/_acnmedia/ Accenture/Conversion-Assets/DotCom/Documents/Global/PDF/Digital_1/ Accenture-Global-Operations-Megatrends-Study-Big-Data-Analytics-v2.pdf   7 Brynjolfsson (2011) Strength in Numbers – How does data-driven decision-making affect firm performance? http://ebusiness.mit.edu/research/papers/2011.12_ Brynjolfsson_Hitt_Kim_Strength%20in%20Numbers_302.pdf  8 www.nesta.org.uk/publications/rise-datavores-how-uk-businesses-can-benefittheir-data   9 Quoted in webinar by Dunhummby, November, 2016. Dunhummby is the division at Tesco responsible for data analytics. 10 www.lego.com/en-us/aboutus/news-room/2011/december/minecraft-projecthits-10-000-votes-on-lego-cuusoo 11 McKinsey & Company, The Big Data revolution in healthcare. www.mckinsey. com/~/media/mckinsey/industries/healthcare%20systems%20and%20services/ our%20insights/the%20big%20data%20revolution%20in%20us%20health%20 care/the_big_data_revolution_in_healthcare.ashx

4 BIG DATA TECHNIQUES AND SOLUTIONS

In previous chapters, you have learnt about data, Big Data, analytics and value creation opportunity and processes. This was more for you to understand value creation from data at a high level. This chapter takes you to the next stage – knowing and understanding the data and importance of value creation will always be associated with the analysis and extraction of insights from the data collected. In this chapter, we will be discussing different techniques of data analytics from old statistical models to the latest predictive analytics and data visualizations.We have mentioned in the previous chapters that data analytics is not a new concept, but ways of performing analytics and extracting and using information from the analytics have been changed and improved drastically – from text analytics, semantics and data visualization to predictive analytics, we have covered some of the most popular techniques in this chapter. Some of you may find this chapter somewhat technical, though we have tried our best to present it in a non-technical and easy to understand way.

Big Data analytics War is ninety percent information. – Napoleon Bonaparte

In the board meeting of a global multimillion-dollar software company based in Central London, its executives were puzzling with a seemingly small but very important question – why was the company recording a consistently declining income in a line of business that accounted for more than 70% of the UK market? The scene in the boardroom was far from unusual. The Sales Director of the company was showing all growing new sales numbers and blaming the product and innovation team for not coming up with top quality products as expected in the market. The Head of Innovation put forward all the recent quality certifications and presented the product roadmap that was agreed on earlier that year – he was delivering as per the plan. The Head of Innovation

56  Big Data techniques and solutions

then suggested that it might be the marketing team not creating the right promotion for the right products that could have caused a decline in revenue in the company. Now, the Sales Director joined the Innovation Head in blaming the marketing team. He considered that the marketing folks were not producing quality leads that the sales team could reach out to and the leads were not ones that would buy high value products. They continued to emphasize that the marketing team was only promoting a newly launched low-price product that was selling in higher numbers but bringing in less revenue. Now it was the CMO’s chance to clarify the situation or to take full responsibility for the declining revenue of the company. Though the CMO was not in agreement with many points in the discussion, he pointed out that all marketing efforts were based on the plan agreed earlier and there was no reason why the sales team should have had issues in bringing more revenue. But he also suggested that there was a possibility that no one in the room had access to the information that could justify the revenue numbers. The CMO requested more details from the IT team – maybe there was something missing in the facts presented and further investigation was needed. The Board decided to give four weeks’ time to the IT head to present the insights, if possible. The IT head spoke to his team of analysts, created a new sub-team for this task and asked them to identify, connect and analyze all the data the company was collecting from different sources and present the report to him in two weeks. The team of analysts approached different teams including sales, innovation, finance and marketing to learn how and where they store different types of data. The first challenge for the newly formed team was that the data were sitting in silos and not ‘talking’ to each other. This is a typical situation in most organizations where integrated data flow among different departments is completely missing. The IT team had to quickly build a platform to bring all the data into one place and started to connect the dots to see if they could spot any information or patterns in different data streams. From telesale text scripts to marketing messages, CRM leads to high value customer spending patterns, all were analyzed. The company then discovered some facts and connected the dots, which started to make sense in explaining the current revenue change in the company. One of the key things that came out of the analysis was that there was a flat fee paid by exiting high value customers to maintain the legacy software systems installed by the company on customers’ servers. Revenue from the maintenance of such systems accounted for 35% of the revenue of the company and most of it was contributing directly to the profit. In the recent past, due to the change in economic conditions, a handful of such high paying customers had stopped using their legacy software systems and started to switch on to low-cost web-based products which were needed as part of their cost cutting initiative. Analysts also identified that the company had introduced a new web SAAS (software as a service)

Big Data techniques and solutions  57

based product early that year and that product was one of the key ones in the current marketing promotion plan. As part of the early customer acquisition strategy for this newly launched product, the sales team was also heavily incentivized for bringing new sales accounts for this SAAS based product. The sales team was striving to grab more new customers for this product that came with a low initial cost, no long-term contract and with no regular maintenance fee. Clearly, the new product had cannibalized existing legacy systems sales and had a big impact on maintenance revenue as well. Another discovery was on the sales team functions where the same sales force that was involved in selling old legacy systems was also selling SAAS product. the telesales team was incentivized for securing confirmed sales appointments and it was easy to grab potential customers’ interest for the new product because of its low pricing and less commitment offer. Sales cycles for the new web-based product were also reduced from six months to one month and customer satisfaction was improved because it was lean, easy to use and a low-cost product with high value. Eventually, total revenue coming from sales declined because most of the calling and selling team was promoting the low-cost new product and even when total numbers of sales were increasing quickly, overall revenue was heavily affected. At the same time, maintenance fees for the old accounts were also decreasing – this change was not necessarily caused by the launch or promotion of the new product but the reduced maintenance revenue was contributing to the reduction in overall revenue. When the findings were presented to the board there were mixed feelings among the board members – instead of blaming each other they now realized that every department was working hard but they were working for themselves instead of working together. There was a need for a collective view approach and long term plan that could deliver value at the company level instead of just a different department level. After several discussions, there was a clear need identified for creating a new strategy that would address not only the existing revenue issue but also help all the parts of the business to work together. The pricing structure for existing customers and legacy systems was redefined to accommodate the recent economic changes and to address their needs with a view to building a long-term business relationship with them. The sales force was divided into two teams – one was focusing on bringing sales for the legacy system and the other one was focusing on selling the newly launched web based lowcost SAAS product. In addition to these, the product development team was involved in the process to decide what features were needed for a customer to move from low cost web products to high cost advance systems and based on their inputs it was decided that some features of the SAAS products had to be dropped and customers with a need for such advance features should be targeted with a high value advance system.

58  Big Data techniques and solutions

The marketing team segmented the market for the new product and started to promote new products to those businesses that could not afford legacy systems and did not need a highly advanced system. The budget for the marketing campaign was further divided for the new product and existing products promotions. When campaign data was recorded, and ROI was calculated in terms of new accounts from SAAS product, then investment in new product promotion increased significantly. Revenue allocation was made more transparent for both the products. KPIs were defined for all the teams in two categories, one for the new product and one for the old product and targets were assigned accordingly. In the next six months, results improved greatly and overall revenue started to grow. Insightful information and key findings came from data analysis in this case which was not new but refined and supported by the data. New recommendations were presented, accepted and implemented by all business heads because they could not only see the pattern but could also connect the dots to make a collective success story. Data (or Big Data) cannot deliver any value in a vacuum – the case discussed above is a clear example of this. The real value of Big Data will be realized when processing of this huge data flow and results of the indepth analysis will start to empower managers and produce fast and effective decision making in an organization. Such evidence-based decision making ensures long-term business growth of a company. Big Data analytics is the process of examining Big Data to uncover hidden patterns, possible correlations and other useful and/or hidden information that can be used to make better decisions. With Big Data analytics, data scientists and others can analyze huge volumes and high frequency of data in a quick and uncomplicated way that a typical oldschool analytics and business intelligence solutions can’t. To understand the difference between the need of Big Data analytics and old-style analytics, let’s assume that your organization is accumulating billions of rows of data with hundreds of millions of data combinations in multiple data stores and abundant formats on an hourly basis and you must run real-time analysis to see fast results to tune the online performance of the store – you cannot achieve this analysis from old-style analytics to a greater degree. High-performance analytics needed for such cases should be processed by Big Data analytics to figure out what’s important and what is not in real-time. Big Data analytics examines enormous amounts of data to uncover hidden patterns, correlations and other insights. Now it is possible to analyze huge amounts of data and produce results in expected forms, almost in real-time – a task that is slower and less efficient with more traditional business intelligence solutions.1

Big Data techniques and solutions  59

Big Data analytics helps organizations harness their data and use it to identify new opportunities. This will eventually result in operational excellence, more competitive products and services, new and innovative products and happy customers. There are three key areas where organizations can create value using Big Data. Operational excellence Big Data technologies such as Hadoop and cloud storage and analytics bring significant cost savings in storing large amounts of data. In addition to that, analytics performed on these data sets eventually help companies in identifying the areas of improvements in different processes that lead to huge cost savings and smooth operations in the organization. Smart decision making Whether it is sales or marketing or strategy, when decisions are supported by data and produced quickly – they result in impressive results for the company. With fast processing capabilities of Big Data infrastructure and available analytics tools, businesses can analyze the data immediately, extracting meaningful information quickly and making fast decisions based on what they’ve learned from the data available with them. This way most of the guess work can be avoided and fast and smart decision making can be seeded in the culture of the organization. Innovation  Unlike before, now businesses have the access and capability to collect customers’ views, feedback and opinion about products or services they are using or value they are expecting from the offerings. Not only this, organizations are now able to identify untold information about their products, services or opportunities and use that in producing new and innovative products or services. A few years ago, a team of IBM engineers identified that the impact of diebetes could be reduced by certain musical therapies and this information was never asked for, talked about or collected formally by anybody in the world – but IBM identified such a pattern in some Twitter conversations and then managed to get all the insights related to this through further research. More information from customers, regarding their needs, feedback or recorded independently is helping companies in producing more innovative products and services.

60  Big Data techniques and solutions

With the ability to gauge customer needs and satisfaction through analytics, comes the power to give customers what they want. Davenport2 points out that with Big Data analytics, more companies are creating new products to meet customers’ needs. As mentioned earlier, data analytics is not a new idea in the world of business but now it is coming from backroom into the boardroom.

Data analytics techniques The price of light is less than the cost of darkness. – Arthur C. Nielsen (Founder of AC Nielsen)

In previous chapters, we have discussed data – structured data, unstructured data and Big Data. Now we can discuss the processing of the data that organizations are collecting regularly in huge volumes from internal sources and from outside – public or partners’ data. The interchangeable use of the words ‘processing’ and ‘analyzing’ is very common as you may be using different tools and techniques for this purpose and different terminology may be used by different companies, but clearly, value from Big Data can be extracted in different ways.There are many ways of storing and analyzing Big Data and you will find that some of the techniques and methods are not necessarily new or dedicated to Big Data as such – fundamental analytics techniques are predefined in statistics and their applications are changing with different cases. As we have new technological solutions (discussed later) and ways to preform different analysis, we will see more techniques and methods being added to the list in the future. There are no pre-defined categories of these analysis techniques; we are trying to put them in a logical order but some of them can belong to more than one category. You may also find that not all the techniques or tools will be applicable for your business. Brief details of these techniques are given here with the view that business managers should have a high-level understanding of these techniques but they may not be involved in the implementation process – their technology team can help in this. On a higher level, you can divide Big Data analytics techniques into the following categories: -------

Statistical analysis Social network analysis Semantics Data visualization Predictive analysis Other analysis

Big Data techniques and solutions  61

Statistical analysis In this information age, data is no longer a rare commodity – instead it is overpowering. The main value lies in the implementation of data storage and more importantly in processing and analyzing the huge volumes of data available to businesses and correctly interpreting its implications. In order to produce all the key information, you may need the right statistical data analysis tools. This kind of analysis is not new to the business world and is an integrated part of classical business intelligence systems but with the emergence of Big Data, it is now possible to create better value derived from the same well-established statistical methods. Correlation and regression These are two of the most popular statistical methods used by businesses for many years to establish the relationship between two dependent or independent variables that are causing or affecting the change in each other. Now with Big Data the accuracy of analysis can be increased further because businesses have more data coming from multiple internal and external sources – that means more information is coming in and businesses have the capability to store and process this data.With the use of correlation, one can understand the relationship between two variables but cannot make changes in any of the variables for experimenting purposes. For example, using correlation it is possible to establish the link between the fact that students with better grades in their schools are more likely to score better in their SAT exams (Scholastic Assessment Tests) but this relation cannot help you identify how good one student should do in their school grades to score particular marks in their SATs. Defining causality using correlation analysis is also not possible. Positive correlation exists when two variables tend to move in the same direction. Sometimes, one of the variables impacts the other. It’s also possible that a third factor or variable influences two independent variables. Positive correlation examples can be found in modern economic theories. For instance, the Law of Demand and Supply enumerates positive correlation between the two.This is one of the first lessons that any student of economics and statistics learns. The Law of Demand states that, all other things remaining constant, in a competitive market, when the demand for a good or service rises, the price rises to the extent that the supply increases and the quantity demanded and the quantity supplied become equal, thus establishing an equilibrium. This is because when demand becomes higher and people are willing to pay a higher price, producers and suppliers start supplying more goods at a higher price. Conversely, when demand for a good or service declines, the price falls, so that the supply falls and

62  Big Data techniques and solutions

equilibrium is established again. This establishes a positive correlation between demand and supply and a cause–effect relationship. An increase in demand is accompanied by an increase in supply and vice versa. However, when it comes to supply and price, there is a negative correlation.When supply decreases, the price rises as more people are competing for the same amount of goods. Consumer spend and GDP or Gross Domestic Product are also positively correlated. When consumers consume more, i.e. spend more, the producers of goods and services produce in higher quantities, causing a rise in GDP.  In the event of a decline in consumer spending, they produce less as compared to the previous period, to keep costs in check as consumers are willing to spend less for the same goods at this juncture.  Regression is a statistical measure used in finance, investing and other disciplines that attempts to determine the strength of the relationship between one dependent variable (usually denoted by Y) and a series of other changing variables (known as independent variables). Regression helps investment and financial managers to value assets and understand the relationships between variables, such as commodity prices and the stocks of businesses dealing in those commodities. The two basic types of regression are linear regression and multiple linear regression, although there are non-linear regression methods for more complicated data and analysis. Linear regression uses one independent variable to explain or predict the outcome of the dependent variable Y, while multiple regression uses two or more independent variables to predict the outcome. Regression can help finance and investment professionals as well as professionals in other businesses. Regression can help predict sales for a company based on weather, previous sales, GDP growth or other conditions. The general form of each type of regression is: Linear Regression:Y = a + bX + u Multiple Regression:Y = a + b1X1 + b2X2 + b3X3 + ... + btXt + u Where: Y = the variable that you are trying to predict (dependent variable) X = the variable that you are using to predict Y (independent variable) a   = the intercept b   = the slope u   = the regression residual Regression takes a group of random variables, thought to be predicting Y, and tries to find a mathematical relationship between them. This connection between variables is typically represented in the form of a straight line. Regression analysis helps in establishing the relationship between dependent and independent variables to analyze and predict the effect on

Big Data techniques and solutions  63

the dependent variable by making changes in the independent variable. There are many business uses of regression analysis. Let’s say that you are leading the sales team in your organization;3 you can predict sales volume and revenue based on the amount spent on advertising and the size of your sales team. Of course, you would have many other independent variables to consider for this analysis in the real world and now with the help of Big Data you would be able to accommodate and analyze a much wider set of independent variables in order to predict sales and redefine your sales strategies. Now you can easily get access to the data, such as what devices your customers are using, and if they are placing an order while connected to Wi-Fi or mobile data. Such information can be valuable for you in taking strategic decisions about launching a mobile app with low data usage features. Figure 4.1 shows four hypothetical scenarios in which one continuous variable is plotted along the X-axis and the other along the Y-axis.4 •• •• ••

S cenario 1 depicts a strong positive association, similar to what we might see for the correlation between infant birth weight and birth length. Scenario 2 depicts a weaker association, that we might expect to see between age and body mass index (which tends to increase with age). Scenario 3 might depict the lack of association between the extent of media exposure in adolescence and age at which adolescents initiate sexual activity.

FIGURE 4.1  Continuous

regression variables plotted on X and Y axis

64  Big Data techniques and solutions

•

Scenario 4 might depict the strong negative association generally observed between the number of hours of aerobic exercise per week and percentage body fat.

Another key use of regression analysis is in optimizing business processes. For example, a call-centre manager can establish the relation between wait time of the callers and number of complaints and work towards reducing the wait time by introducing new processes or resources. A baker can establish the relationship between shelf life of a cake and oven temperature and can adjust the temperature to give a longer life to the cake he is baking. Big Data brings a lot of data points that can present huge analysis and improvement opportunities for a business. A/B testing (statistical hypothesis testing) A/B testing is an old technique for randomized experiments used by businesses and statisticians. In a controlled environment, businesses run experiments on more than one variation with a metric that defines success and analyze the results. In other terms, A/B testing consists of taking two comparable groups of users and exposing them to two different versions (the control and a variation) of a product experience. By doing that and measuring how each of the groups interacts with the product we hope to infer which of the two versions best serves its purpose. Many retail banks use this technique to optimize their product offering. Hypothetically assume that you are leading a team of a retail bank that is going to launch a new credit card product and want to see how your customers will respond to a little variation in the interest rates and annual fee. Say you sent 5,000 credit card offers to your prospective customers in two variations, one with an interest rate of 10.7% and an annual fee of £54 and other with an interest rate of 13.7% and no annual fee – based upon the response from your customers you can optimize your offering and rollout the winning product (winning variation of the split test) to all your targeted customers. Banks like Capital One are using this technique extensively and as published by a report in 2007, the bank has run 28,000 separate tests on current and prospective customers.5 A/B split tests are very popular among technology companies and in many cases it is (wrongly) assumed that A/B testing is only applicable on websites, emails or system usability. Many companies are using A/B variations on their website to identify the right version that can help them achieve their goals. Kiva.org is a charitable organization that wanted to increase donations on its website and decided to include more content on the home page

Big Data techniques and solutions  65

to drive confidence in visitors. the experiment was run on two versions of the home page – the existing page and a new page with a text box appended to the end of the page. The new text box had some key components – frequently asked questions, testimonials and statistics on the website usage. The new version has helped Kiva increase donation conversion by 11.5%.6 Almost anything on your website that affects visitor behaviour can be A/B tested. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12.

Headlines Sub headlines Paragraph text Testimonials Call to Action text Call to Action button Links Images Content near the fold Social proof Media mentions Awards and badges

Advanced tests can include pricing structures, sales promotions, free trial lengths, navigation and UX experiences, free or paid delivery and more. A/B tests can also be run on other marketing communication platforms like emails. The A/B testing feature in an email is a way to let you send two slightly different emails to your audience so that you can obtain information on which triggered the most engagement and then use the most effective email to send to the rest of your list.  Conversion Voodoo, an online conversion consulting company has run an experiment for one of their clients, by changing the subject line of an email to see how readers will respond to the content of the email. They have tested three variations of the promotion email with three different subject lines and the results7 can be seen in Figure 4.2. Clearly emails with the subject line ‘Merry Christmas %FIRST_NAME%’ have around double the open and click rates. So presumably you are sending a Christmas promotion email to 100,000 customers and you know what the subject line should be, based on your A/B test, then you will have a chance of much more responses and conversions than otherwise. Apart from these, there are a few simple statistical analysis methods including mean, median, mode, standard deviation and sample size determination8 etc. –

66  Big Data techniques and solutions

FIGURE 4.2  Email

open and click rates change based on their subjects

Mean The arithmetic mean, more commonly known as ‘the average’, is the sum of a list of numbers divided by the number of items on the list. The mean is useful in determining the overall trend of a data set or providing a rapid snapshot of your data. Another advantage of the mean is that it’s very easy and quick to calculate. In some data sets, the mean is also closely related to the mode and the median (two other measurements near the average). However, in a data set with a high number of outliers or a skewed distribution, the mean simply doesn’t provide the accuracy you need for a nuanced decision. Pros: It works well for lists that are simply combined (added) together. Easy to calculate: just add and divide. It’s intuitive – it’s the number ‘in the middle’, pulled up by large values and brought down by smaller ones. Cons: •• The average can be skewed by outliers – it doesn’t deal well with wildly varying samples. The average of 100, 200 and -300 is 0, which is misleading. •• •• •

Median In a given list of items, median is the item that appears in the middle of the sorted list. Data such as housing prices are always presented in median because it gives you an idea of the middle band of the overall data set. Pros: Handles outliers well — often the most accurate representation of a group. • Splits data into two groups, each with the same number of items. Cons: •• Can be harder to calculate: you need to sort the list first. ••

Big Data techniques and solutions  67

Mode In a given list of items, Mode is the most popular or the biggest item. Mode can be used for including one particular feature in the product on the basis of customers’ feedback. You will consider all the feedback coming from customers, but it may make sense to pick the most popular feature for implementing first. •• •• • •• •

Pros: Works well for exclusive voting situations (this choice or that one; no compromise). Gives a choice that the most people wanted (whereas the average can give a choice that nobody wanted). Simple to understand. Cons: Requires more effort to compute (must tally up the votes). ‘Winner takes all’ — there’s no middle path.

Standard deviation The standard deviation, often represented with the Greek letter sigma, is the measure of a spread of data around the mean. A high standard deviation signifies that data is spread more widely from the mean, where a low standard deviation signals that more data align with the mean. In a portfolio of data analysis methods, the standard deviation is useful for quickly determining dispersion of data points. Just like the mean, the standard deviation is deceptive if taken alone. For example, if the data have a very strange pattern such as a non-normal curve or a large number of outliers, then the standard deviation won’t give you all the information you may need. Consider two small businesses with four employees each (see Figure 4.3). In one business, two employees make $19 an hour and the other two make $21. In the second business, two employees make $15 an hour, one makes $24 and the last makes $26.9 In both companies, the average wage is $20 an hour, but the distribution of hourly wages is clearly different. In company A, all four employees’ wages are tightly bunched around that average, while at company B, there’s a big spread between the two employees making $15 and the other two employees. Sample size determination When measuring a large data set or population, like a workforce, you don’t always need to collect information from every member of that population – a sample does the job just as well. The trick is to determine the right size for a

68  Big Data techniques and solutions

FIGURE 4.3  Employees’ earning

plotting for Company A and Company B

sample to be accurate. Using proportion and standard deviation methods, you are able to accurately determine the right sample size you need to make your data collection statistically significant.When studying a new, untested variable in a population, your proportion equations might need to rely on certain assumptions. However, these assumptions might be completely inaccurate. This error is then passed along to your sample size determination and then onto the rest of your statistical data analysis Social network analysis Also known as SNA, social network analysis is popular in Big Data technology for extracting insightful information from unstructured data. With the emergence and popularity of social media apps such as Facebook, Twitter, Instagram and Snapchat etc. new opportunities are being introduced to companies. Social network analysis (SNA) is the mapping and measuring of relationships and flows between people, groups, organizations, computers, URLs and other connected information/knowledge entities.10 The nodes in the network are people and groups while the links show relationships or flows between the nodes. SNA provides both a visual and a mathematical analysis of human relationships. In the field of management consulting, this methodology is called Organizational Network Analysis (ONA). ONA allows you to X-ray your organization and reveal the managerial nervous system that connects everything. Social network analysis is used widely in the social and behavioural sciences, as well as in economics, marketing and industrial engineering. The social network perspective focuses on relationships among social entities and is an important addition to standard social and behavioural research, which is primarily concerned with attributes of the social units. This is the analysis of how two or more users/people relate to each other and what activities they are involved in. In terms of SNA all the connected

Big Data techniques and solutions  69

FIGURE 4.4 Typical

graph of mutual friends on a social network

users are called a ‘node’ in a social network or graph – and data scientists can establish the logical connections between these nodes. The image in Figure 4.4 represents a network of nodes in Facebook – where you can identify information like common nodes, influential nodes and active nodes/ users etc. Figure 4.4 represents a graph of mutual friends.11 This kind of information can be very useful in different aspects of the business. In 2013 a Middle East-based telecommunications company, Qatar Telecom, was going through changes in their corporate branding and they wanted to engage with their existing customers and utilize their social network to spread the news about their new brand. They hired a London-based social media analysis company to help them identify the key influencers on social media in their market and decided to approach them to engage in spreading their new branding message. They prepared a pre-launch strategy

70  Big Data techniques and solutions

and the social network influencers worked with them in spreading the word about the new brand before the company made a formal announcement. Pre-launch analysis and work in coordination with the launch event resulted in a successful brand transformation for the company.12 Some years ago, LinkedIn introduced a very effective feature called ‘People you may know’ – it was a recommendation engine based on the social graph of a user, its direct and indirect network and places he had studied or worked in the past etc. Based upon this SNA powered feature, LinkedIn can accurately suggest to its users people they may know and could manage to get mileage from in terms of user acquisition and engagement.13 Interestingly Facebook has also implemented a similar feature based on social network analysis and started to identify and recommend friends to its users by its ‘People you may know’ feature, but unlike LinkedIn, Facebook is using your phone data and wrongly connecting its users to others on the basis of both having one common phone number that can be the number of any service provider. On that basis many clients of a therapist can start to appear in each other’s recommendations. For example, if you search ‘people you may know therapist’ on Twitter – you may find many people complaining about wrong recommendations by Facebook. Analysis of activities and content Another important aspect of SNA is tracking data about the activities and content generated by users on social media. User activities and behaviour are used by businesses in understanding their customers better and predicting their future buying patterns and activities. Facebook advertisements are one of the best examples that work along these lines. If a user is sharing content and talking about newborn babies and is interested in baby products, then it is more likely that they may be new parents or associated with someone who is or is going to be a parent and this information can present potential opportunities to the businesses targeting new parents. Recently Facebook has added one new category under the demographic filters in its advertising modules with the category ‘Expectant Parents’ – clearly not all the users are updating their profiles with their expectant status, but Facebook is using their activities and postings and guessing their state and adding them to the Expectant Parents’ demography (see Figure 4.5). Businesses are not only using these behavioural insights to promote and sell their products, but they are also implementing these learnings to create and innovate new products and features. Social media has presented a new way for people to communicate openly and that is presenting huge opportunities for businesses.

Big Data techniques and solutions  71

FIGURE 4.5  Example

advertisement targeting on Facebook

Businesses are also analyzing activities outside the social networks and then using that to reach their customers on social networks. If you are planning a holiday and looking to buy some related items such as flight tickets or a hotel etc. then you may start to see many advertisements appearing on your screen from suppliers after a few visit to some websites. These advertisements appear based on your visit to one such website that leaves a small piece of code on your computer – this piece of code is called a ‘cookie’ – and then suppliers start to show you advertisements in real time on websites like Facebook that is one of the most popular portals right now. Analysis of the content generated by use of social media is very useful and I have covered the details in the next section on semantics.

72  Big Data techniques and solutions

Semantics Semantics is the branch of linguistics and logic concerned with meaning.The two main areas are logical semantics, concerned with matters such as sense and reference and presupposition and implication, and lexical semantics, concerned with the analysis of word meanings and relations between them. In this context, we are mainly talking about lexical semantics. Social media and the democratization of the Internet have given birth to various platforms for people to participate in discussions or unilaterally express their thoughts – from forums and blog posts to Facebook and Twitter, users are highly engaged in participation and are generating huge amounts of text and other data. These developments have triggered greater interest in areas such as text mining and analysis to extract meaning and information from huge amounts of text without reading them manually. in particular, Real Time Intention (RTI) has started to gain a lot of popularity – RTI can be defined as ‘a text expression signifying an intent to perform an activity in the near future’. On the basis of identified intent, businesses can consider pushing products, discounts and offers to targeted customers. We have many such examples, like Tesco’s Clubcard and Amazon’s recommendation engines where such intentions are identified on the basis of past buying patterns, but now on the basis of what you are writing on social media, you may be presented with targeted offerings. Online  unstructured  text can be  processed and  classified into one of many predefined categories of general human intentions – interests, questions, enquiries, objections, complaints  and so on. While researchers have highlighted the ‘huge opportunity towards web personalization’ and ‘Dynamic user context’14 from text  analytics, understanding RTI could unlock even greater business benefits. This method also helps banks and other financial institutions, where most of the email exchanged between employees and suppliers may contain confidential and sensitive information. From the security perspective, many financial institutes have implemented intention analysis layers inbetween the sender and receiver of the emails to make sure the information is secured and encrypted before it goes out of the organizational boundaries. Financial institutes like HSBC have also used RTI in replacing a big part of their support staff with automated chatting bots. On the HSBC website you will find a chat box where you can type any question you want to ask, and they use intention analysis technology to identify pre-defined and classified answers to you. This way most users get answers to their questions and avoid making any call to the call centre, thus saving money for the bank. There are many organizations which use such intention-based chatting functionality on their portals.

Big Data techniques and solutions  73

Semantics analysis technology helps businesses to extract real value from the heap of  unstructured  text coming from different platforms  including email, blogs, forums and social media etc. Identifying the intent and other key information from social conversations may help businesses in innovating and improving their operational processes, increasing their top line revenue and building business strategies.  Text analytics There are huge amounts of unstructured data being generated on a daily basis in multiple places including websites, social media portals and log files. Most of the data come in an unstructured text format and companies are using many techniques, including pattern matching, clustering and classification to understand this huge amount of text data. Text analysis is about deriving high-quality structured data from unstructured text. Another name for text analytics is text mining. A good reason for using text analytics might be to extract additional data about customers from unstructured data sources to enrich customer master data, and to produce new customer insights or to determine sentiment about products and services. There are many uses of text analysis – some of them are listed below: ••

••

••

••

 ase management – from insurance claims assessment, healthcare patient C records to crime-related interviews and reports, there is a lot of text accumulated – and it is a tedious task to read through all the text repeatedly on a need basis. Text analysis can help in summarizing and identifying important and key points. Competitor analysis and market research – market analysis, survey and competitors’ analysis etc are not just limited to pre-defined factors associated with the business. Information about the market and industry is available in different forms at different places including digital and manual. If all the data can be converted to digital data and associated with other formatted data sets then a combined text analysis can be highly useful and will bring hidden insights. Media coverage analysis – most PR firms use text mining and analysis processes in identifying their clients’ media mentions and coverage so they can react accordingly to the same. Google Alert is one such tool that regularly mines through the web and brings media mentions of the brand or keyword to the company. Customers’ voice – entity extraction (the parsing and extracting of entities from raw text) is a key part of text analytics. Some of the examples of entity extraction from regular unstructured text can include following: ºº Company names

74  Big Data techniques and solutions

ºº ºº

Dates and times Product names etc.

In many cases, entity extraction from raw text can be converted into an automated entity recognition where text can be parsed and classified. Entities are automatically selected from the text by the software. This is a very common requirement from businesses that are taking unstructured data seriously and have started to collect and work on raw data such as social media feeds or machine logs. In the previous list of uses, recognizing dates, times, monetary amounts and so on may be something that text analytics software can do out of the box, without you having to help figure out what these extractions are. There are obvious benefits of such an approach. Recently with the help of new Big Data storage and analytics tools, the time required to perform entity extraction has drastically reduced and it can be scaled across large volumes of text very quickly.The outcome of text analytics entity extraction is a group of structured data that can be merged with enterprise data for further analysis. If there are known patterns such as email or phone numbers appearing in the text documents, then text analytics engines can easily identify and classify the content of documents.Text analysis can also be used for the density detection of words (often represented as a word cloud) and can identify the relations and connections with other words in the document. Business-specific patterns and entities can also be defined so text analytics engines can classify extracted words and sentences and keep them in the defined categories. A typical social media or web-based text analytics engine reads and curates social and web conversational data in the form of text, images, video etc. using techniques like web scraping or accessing predefined social media data APIs such as GNIP. The engine then builds  taxonomy-based classification based on initial  learning  data and rules defined by the chosen industry or vertical. Classification algorithms initially identify and classify text into categories using training sets of data and then progressively refine these classifications on the basis of new data and large volumes of data processing. It is important to understand that semantic analysis is not just about word identification and classification – context before content is very important in the analysis. Context is identified using multiple taxonomies and connected ontologies. So, one keyword appearing in two different sentences will have totally different intentions or sentiments classified by the system, for example:  •• •

This TV is large => holds Positive sentiment, as people like the large size of a TV  This mobile is large => holds Negative sentiment, as people don’t like large mobiles 

Big Data techniques and solutions  75

So, when a keyword ‘large’ appears under the size node of a TV taxonomy it is classified as Positive and when it appears under Mobile taxonomy it is classified as Negative – this is based on the predefined training data and classification takes place accordingly. Context identification is not just based upon related and previous conversation; it can also be identified using other signals such as profile information of the person, their location, device and time of the conversation. A separate taxonomy can be defined for these classification signals.  Sentiment analysis Sentiment analysis is the process of determining whether the meaning of the statement written in a language can be considered positive, negative or neutral. It’s also known as opinion mining, deriving the opinion or attitude of a speaker. A common use for this technology is to discover how people feel about a particular topic. An analytics engine parses  sentences  and phrases  based on defined taxonomies and ontologies to identify the context and meaning of the statement. Understanding the context of the conversation is important in order to understand, classify and act on the meaning of the statement presented in the given text and associate sentiments with that text. There are millions of conversations taking place on social media on a daily basis and it is important for businesses to understand what their consumers feel about them, so they can improve their offerings, resolve consumer issues and gain trust. Sentiment analysis identifies the associated sentiments in a conversation and empowers brands to gain insights from it. Say you want to know if people on Twitter think that Chinese food in San Francisco is good or bad. Twitter conversations’ sentiment analysis may answer this question.You can even learn why people think the food is good or bad, by extracting the exact words that indicate why people did or didn’t like the food. If too salty shows as a common theme, for example, you immediately have a better idea of why consumers aren’t happy. Sentiment analysis can also be used for online review segregation, classification, summary and in identifying fake reviews posted on websites. Using sentiment analysis all feedback can be summarized in an easy to read short report that can be referred to by the customer before making a purchasing decision. There is another challenge, in that the reputation and sales of businesses can be affected negatively if a number of fake reviews are being published – such things can be done by a competitor or out of fun – now sentiment analysis technology can be used to identify those fake reviews, so businesses can get them removed and safeguard their position.

76  Big Data techniques and solutions

Marketing and sales intelligence  Many businesses and market research firms are still using the same tools, techniques and processes for data collection and analysis as they were doing 10 years back. Focus groups, surveys and passive monitoring all need to change in light of the new technological inventions and behavioral changes in consumers today. Methods such as surveys should be shorter and mobile friendly etc.; these are a few tactical issues that are eventually changing. But in addition to this there are huge amounts of transactional data stored inside businesses and these data hold very valuable information about customer behaviours – this is very important for businesses in understanding the market and changes in the ecosystem. The answers to many of the questions that are part of typical market research could already be within that data, and easily analyzed and used to the advantage of businesses. Businesses can use not only market research also unstructured data available on social media and other web conversations and run that through the analytical engine to identify the potential target audience and buyers (sales leads) of  products and services. There are many industries, from web development to software selling, where such analytical approaches are being used to generate sales leads. People talk about many products and services on social media – these conversations can be classified in real-time and can be presented to others who are looking to buy a product or service, so they can make an informed decision. As with some applications, Big Data analytics engines are also powering open product review portals where data is coming from the public domain and opinion is built automatically using text analytics – this presents a very transparent and smart way of using product reviews for consumers in any industry. There are many applications built on Big Data analytics and classification to understand hidden insights from social media conversation and unstructured text analysis to  perform competitor analysis, go-to-market strategy study, demand–supply analysis etc.  Data visualization A picture is worth a thousand words Data visualization presents data in a graphical format. It makes decision ­making easy, as you are able to see the analytics/findings presented visually. Instead of struggling with numbers you can focus on clear insights, can grasp difficult concepts or identify new patterns in no time. Now it is also possible to have interactive visualization of the data where managers can drill down into charts and graphs for more details and make changes in the runtime to understand the patterns.

Big Data techniques and solutions  77

Data visualization is a very powerful way of understanding and using data analytics – and you can implement data analytics in any part of your business, from top line sales improvement to bottom line operation efficiencies. Managers have been using Excel sheet-based graphs and charts for a long time. But with the introduction of innovative charts and graphs by companies such as Qlik and Tableau, data visualization has reached the next level. Story telling with the help of innovative designs and data visualization is known as Infographics – this is used by many consumer-focused businesses to present information to the open world.There are many different graphs, which can help present information hidden behind complex data sets in very effective ways. In most businesses, data visualization can add a lot of value in different ways, but here are some key areas where it can be used in most businesses: ••

 clear data visualization can help businesses in identifying the key areas A of improvements based on their performance and expected return. It can not only help by improving or giving attention to low performing areas, it also helps you identify those areas that are performing better, and more resources can be deployed to them for better results.

Performance can also be visualized against a timeline to do a comparative study. Please see Figures 4.6–4.8 where Economic performance of the Amsterdam Metropolitan Area is depicted by sector and years.You can see the charts and changes for three years – 1995, 2005 and 2009.

FIGURE 4.6  Economic

performance of the Amsterdam Metropolitan Area in 1995

78  Big Data techniques and solutions

FIGURE 4.7  Economic

performance of the Amsterdam Metropolitan Area in 2005

FIGURE 4.8  Economic

performance of the Amsterdam Metropolitan Area in 2009

••

 ith the help of dynamic data visualization, it is possible to identify the W factors that influence customer behaviours and buying patterns. This is also possible through regression that we discussed earlier in this chapter, but with the help of dynamic visualization, a manager can just drag the value of one factor to see the change in other factor in real time.

FIGURE 4.9  Life

expectancy by country in 1850

FIGURE 4.10  Life

expectancy by country in 1950

FIGURE 4.11  Life

expectancy by country in 2000

80  Big Data techniques and solutions

See Figures 4.9–4.11 where change in years income per person is reflected in people’s life expectancy.15 ••

Predicted sales is another interesting use of data visualization because it

can present a collective view of sales and all factors affecting sales in one screen and a manager can take quick and smart decision by looking at only one screen – see Figure 4.12.16 The Guardian newspaper in the UK is known for presenting complex data and information in very easy to understand info-graphics using publicly available data sets. For example, Figure 4.13 shows the UK government’s spending and earning in 2014 in one view presented by The Guardian.17 Figure 4.14 gives another visual, this time showing net migration during the past five years. There are many visual alternatives but the basic principle for browsing and searching might be summarized as a ‘visual information seeking mantra’: overview first, zoom and filter, then details-on-demand.  There are many powerful visualization tools available now and many of them are already part of popular visualization software like Tableau or QlikView. Here I have briefly included a small list of some popular and basic visualization charts that are used in different forms to present the information obtained from data sets.

FIGURE 4.12  Example

sales forecast display on line graph

FIGURE 4.13

UK government spending and earning in 2014

FIGURE 4.14

Net migration during the past five years

82

Big Data techniques and solutions

Time series graphs/line charts Line charts are one of the most popular and basic charts that display information about multiple parameters and/or variables as a series of data points connected by lines of different shapes. In many cases the horizontal axis of the chart represents time where changes are depicted against a time variable – these are also known as time series graphs. Time series graphs are used to identify the pattern changes during a set period of time – they also represent the time of seasonal and irregular movements or fluctuations of activities/information during the period analyzed. Multiple variables can be also plotted and analyzed during the same time period. Figure 4.15 presents President Obama’s approval rating during 2013 based on the data collected from three independent polls – we can clearly observe changes in the pattern here.18

FIGURE 4.15

President Obama’s approval rating during 2013

Big Data techniques and solutions 83

FIGURE 4.16

Heat map representing the temperature in mid-2012 across the USA

Heat maps A heat map is a graphical representation of data where variables or individual values are presented in different colours in a two-dimensional chart. One value from the data set, such as revenue, represents the size and a value, such as % change, determines the colour. Tree maps, geographic maps, histograms, grids etc. are few popular heat maps. The heat map example given in Figure 4.16 represents the temperature in mid-2012 across the USA.19 Bubble chart A bubble chart represents three dimensional data visualization where two values represent the position of the data point (x,y) and their value represents the size of the bubble (radius of the bubble). In some cases, a bubble chart can be considered as an extension of scattered charts where density or size is provided as third value. Figure 4.17 visualizes the sales of an organization in different categories using a multi-coloured bubble chart.

84  Big Data techniques and solutions

FIGURE 4.17 An

chart

organization in different categories using multi-coloured bubble

Predictive analysis The Department of Transport in the UK organized a ‘hackathon’ in 2013 in Newcastle, UK. If you are not familiar with the term hackathon, it is a shortterm event for programmers to come and build a minimal viable product or prototype around an idea or solve a problem presented by the organizers. In the hackathon, an interesting problem was presented to programmers – they were given access to open transportation and traffic data and asked to prepare a model to predict activities around public transportation in the UK. A group of smart engineers worked over night and developed a system using this transportation data to avoid routes with heavy traffic in peak hours of the day. They learnt from the existing data about the pattern and locations of busy and toll roads and based on some other factors they could manage to predict the best routes for drivers to take. Some similar functionalities are visible when you search for a route on Google Maps and it will suggest a couple of routes and timings based on real-time data analysis. What these systems are doing is analyzing past data and recognizing patterns and applying

Big Data techniques and solutions  85

this information to the real-time data to identify what will happen next and suggesting appropriate options to the user. Google’s auto complete search or auto correct spellings – ‘Did you mean …?’ – feature is built around the data analysis of past users, Facebook’s suggestions of friends is based on the analysis of a user’s associations with other users via different institutes, activities or other friends. Similar to Facebook, LinkedIn has its own predictive algorithm. One of the most popular recommendation engines was built by Amazon on the basis of data and behavioural analysis of past customers. On the basis of previous purchase patterns, Amazon’s recommendation engine predicts the items that a user is most likely to buy. Predictive analytics is the holy grail of Big Data analytics and there are a lot of techniques and algorithms that are written to achieve this in different companies and verticals.

Summary In previous chapters, we have seen the importance of Big Data analytics and how value creation is one of the most important goals of any business. In this chapter, we have extended our learning on Big Data analytics by discussing several analytics techniques, tools and examples that can help practising managers to identify hidden insights in the data and make best use of that in creating value for their organization. These are a very few important tools and techniques that help you in most of the Big Data analytics projects and we have covered them here with a view to developing a good understanding of their usage and viability in different cases. Every technique is different, and one may be more useful for your purpose than another. These techniques help us in deriving the patterns and analysis from the data generated and collected. When you are executing a Big Data project and you don’t think that you have access to all the sorts of data you may need, then you can analyze some of the tools and external proxy data to run an analysis. You may not need to use all the techniques or tools yourself or implement them directly, and your technical team will always help you in implementing these tools but practising managers should be able to know and discuss these in order to achieve their project goals effectively. When you have learnt about Big Data, analytics, value creation and you are equipped with some important tools, now is the time for you to know about the most important takeaway of this book – the C-ADAPT model, which we will cover in the next chapter.

Notes  1 www.sas.com/en_us/insights/analytics/big-data-analytics.html  2 www.tomdavenport.com/

86  Big Data techniques and solutions

 3 http://abyss.uoregon.edu/~js/glossary/correlation.html  4 h ttp://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704_Correlation -Regression/BS704_Correlation-Regression_print.html  5 www.slideshare.net/amdiaweb/how-direct-marketing-applies-in-a-multichannel-marketing-world  6 www.conversionvoodoo.com/blog/2014/01/conversion-case-study-kiva-org/  7 w ww.conversionvoodoo.com/blog/2013/12/merry-christmas​-happy-holidays-choose-wisely/  8 https://betterexplained.com/articles/how-to-analyze-data-using-the-average/  9 http://uk.businessinsider.com/standard-deviation-2014-12 10 www.orgnet.com/sna.html 11 http://mashable.com/2009/08/21/gorgeous-facebook-visualizations/#_ MNwQR9j1EqI 12  Co-author of this book, Atal Malviya, was involved in the project. 13 www.zdnet.com/article/how-linkedins-people-we-may-know-feature-is-so-accurate/ 14 https://ieeexplore.ieee.org/abstract/document/5999514 15 www.gapminder.org/tools/#_state_marker_size_zoomedMin:15&zoomedMax: 1400000000;;;&chart-type=bubbles 16 https://exceldashboardschool.com/sales-forecast-chart/ 17 www.theguardian.com/news/datablog/2014/mar/21/budget-2014-tax-spendingvisualised 18 www.businessinsider.com/obama-approval-rating-polls-rcp-average-201312?IR=T 19 www.livescience.com/19218-record-high-temperatures.html

5 INTRODUCING THE MODEL Design and implementation We have already seen in previous chapters, it is an undeniable fact that Big Data is a huge opportunity for practising managers to create financial and non-financial value for their organizations. We have also covered a few tools and techniques and presented some examples for any practising manager to learn and move ahead on this opportune route. Irrespective of industry, vertical or domain, presence of data and associated opportunities are available in all directions. But even after being aware of this, many managers find it difficult to work on this opportunity and conceptualize their approach toward value creation – financial or non-financial. Often non-technical business mangers find themselves highly dependent on technology teams or professionals when projects like Big Data start to have an effect in an organization. Fundamentally, this dependence creates big challenges and barriers for managers wishing to work on such opportunities in the best way possible. In Chapter 4, we discussed an example of a software company based out of London – in most organizations, the situation with non-technical managers will be the same as it was for the CMO and Sales Director in that example. Considering such situations in businesses and knowing that managers cannot go for highly technical learning in a short span of time, we could clearly see the need for a much simpler framework of approach for practising managers who are planning to work on Big Data projects. A systematic model and process for managers to think about building strategies for Big Data projects will empower them with great strengths. Along with the technical or non-technical parts of the value-creation process we should always be aware of how we are going to measure the value created and, as suggested in Chapter 3, that the difference of the revenue generated and cost of revenue generation including all processes such as Big Data setup and maintenance should be positive – otherwise even after using

88  Introducing the C-ADAPT model

the best possible Big Data setup and team, and showing that top line revenue has improved, the organization will eventually not create the value that can positively contribute to the growth of the company. Practising managers should consider this testing or measurement part as very important because if the overall value created by the Big Data analytics (or any other project) is negative and the process is iterative and repeated then in the long run the organization will face accumulated big losses that can pose a big threat to the future of the company. While working on many projects in the past, and dealing with many practising managers, we realized that it is possible for managers to have a systematic model and framework to identify challenges or opportunities and address them using Big Data analytics irrespective of the vertical or industry they are in. Although we have already presented many techniques that can be used by managers and teams in executing Big Data analytics projects, it is necessary to put in place a proven framework. Based on our previous work and observations from many Big Data case studies, we now present you with a model that can help any practising manager to lead Big Data projects. the last stage of the model will help you measure the value created to ensure that the whole Big Data project is being executed well. In this chapter, we present you with two key takeaways: the C-ADAPT model and the C-ADAPT worksheet. 1. C-ADAPT model of Big Data value creation.  This comprehensive model was developed for strategic practising managers to help identify opportunities and challenges, define the project with clear goals and execute them successfully. This model is put together in a very simple and easy to use way so anyone from any part of business can not only understand it quickly but also implement it in any Big Data related project. It is an iterative process model with six key stages – we will be discussing the model and its use in detail in coming sections. The model encapsulates the key parts of a Big Data project in a way that a practising manager can start the project with high level goals and end the project with a very clear measurement of the value created and then decide if any changes or improvements are needed for next iterations – but it is important that we keep an account of all the financial and non-financial costs before we reach the final Testing stage. 2. C-ADAPT worksheet of Big Data value creation. As the name suggests, it is the worksheet that managers use to systematically define different complements of the project and map them with

Introducing the C-ADAPT model  89

distinct parts of the suggested C-ADAPT model in order to execute the project successfully. The worksheet can also be used as a practice instrument to hone the skills required to deal with complex data projects. We also encourage you to use this worksheet to work on any past project or case studies to understand the model and processes quickly. We have also included some case studies in the next chapter where you can try to map the model with executed cases to understand how success can be achieved in any type of project. Before we move ahead and start to work on the model and worksheet, we should remember that when we are presented with case studies of an external organization and Big Data success – it mostly talks about the goal achieved as being defined before the project started – in most of the cases, the final calculation of the value created will be missing because we cannot have internal (mostly financial) data to measure the value created. But when you, as a practising manager, execute any such project in your own organization you should use the calculation and methods like NPV etc., given in Chapter 3 to ensure that your project is ultimately creating positive financial value for the organization and is not just a hobby project for you.

C-ADAPT model of Big Data value creation The C-ADAPT model was developed by us after working with many small and large clients on their Big Data analytics projects. This is a systematic approach that any manager or business leader can follow to extract the best out of their Big Data opportunity or to address any challenges. This is an iterative model where a manager should follow six pre-defined stages in any project and then he may test and repeat the same stages for improving the results produced in previous iterations. The name C-ADAPT is made of the first letters of the six different stages of the model: 1. 2. 3. 4. 5. 6.

C: Challenges and/or goals A: Areas identification D: Data discovery A: Analysis and insights P: Presentation and visualization T: Testing the value created, refine and repeat

C-ADAPT is an iterative sequential model whose logical flow is shown in Figure 5.1.

90  Introducing the C-ADAPT model

1-C Challenges & Goals

6-T

2-A

Testing VC, Refine & Repeat

Areas identified

5-P

3-D

Presentations & Decisions

Data Discovery

4-A Analysis & Insights

FIGURE 5.1  Logical

flow of iterative C-ADAPT model

1. C (Challenges and/or goals) This is the first stage of the model where C represents the challenges – a list of key issues, problems or challenges that need to be addressed by the organization. This may also have a list of new goals or opportunities that the company is willing to create or address. Clearly this stage suggests that on one hand we can use this model to address issues in existing projects or processes, and on the other hand we can use this model to create new strategies for achieving new business goals with innovative outcomes. For practising managers, it is important to identify the challenges that need to be addressed or goals that need to be achieved before moving ahead with the other parts of the project. At this stage the whole team can come together to prepare a list of goals if possible. In some cases, it is just the manager who is going to define the goals at this stage – and that is okay too. To understand the model better, let us look at a fictitious example. Assume that you are heading the sales department of an e-commerce company called E-COMM that is selling electronics products in your home country – the United Kingdom. The E-COMM web portal is running on an existing plug

Introducing the C-ADAPT model  91

and play e-commerce system such as Shopify or Magento that provides an easy way of managing products, payments etc. from the admin sections of the website.Your primary goal is to increase the profitability of the company. You and your team should come up with a plan and execution strategy and implement it for the best results possible. The first step is to identify the key goal (or set of goals) of the project and mark them as the ‘C’ of the C-ADAPT model. It is interesting to note that there are many factors that can have a final impact on the profitability of the company – it may be top-line revenue factors or the bottom line. Assuming E-COMM is making £5 million profit every year and the team decides to run a Big Data project that costs £3 million, then we should wait for E-COMM to make at least £8 million before classifying it as a successful project. We will cover this calculation in the sixth stage of the C-ADAPT model, testing the value created, refine and repeat. These are the Cs (challenges) of the project: Increase the profitability of E-COMM. 2. A (Areas identification) The next stage of the model is ‘A’, which represents the areas of the business which the challenges or goals identified in the last stage are related with or can have an impact on. These areas can be related directly or indirectly with the list of Cs – and in some cases, you will need to do some work before you can identify what the hidden areas are that may be affecting the final goals. This list of areas can also be prepared on the basis of assumptions and with the help of some brainstorming among the other managers, business heads and other team members. The longer the list, the better it is for you, as you can drop some areas later in the stage if you find them irrelevant based on your findings. Mapping this with our project, where the goal is to increase the profitability of E-COMM, we can consider the following ways to achieve the goals as initially discussed in Chapter 3. We discussed that increasing 1% on the price of a product can give 11% rise in profit – by far, increasing the price is one of the most effective ways of increasing the profit. Similar to that, we can also consider other options for creating value – here is the list of factors that we discussed in Chapter 3 which can have an impact on profitability: • • • •

1% increase in price – 11% increase in profit 1% reduction in variable cost – 7.8% increase in profit 1% volume increase – 3.3% increase in profit 1% cut in fixed costs – 2.3% increase in profit

92  Introducing the C-ADAPT model

By increasing the price of the product we are selling, we can achieve higher profitability as the margins will go higher with each transition; this may have an impact on the number of items sold because higher prices may deter some customers. Competitors can also take advantage of this move and aggressively market their products on price point or will see this as an opportunity to increase their product price as well. On the other hand, another approach could be to work on the cost side, especially if the market is competitive and price sensitive. Reducing the cost of the production or procurement and logistics can help you reduce the overall cost of product sold because most of the data and processes are available internally in the organization – negotiating a better price with the product or service suppliers, from raw materials and assembly parts to courier and delivery, can help you increase the profitability. This way, because you are not touching the price of the product and no external impact would be visible, your competitors cannot use this change as an opportunity for them. Table 5.1 offers a recap of the key differences of the value drivers and associated levels of uncertainty. You can also think about process optimization or operational excellence in order to shorten the production cycle or reduce the waste in the process. If the cost side can be optimized, then sale numbers of items sold will collectively TABLE 5.1 Value drivers and creation hierarchy

Value driver

Value creation hierarchy

Level of uncertainty in value capture

Revenue

Unpredictable and hard to quantify

Costs

Predictable and quantifiable

Working Capital

Predictable and quantifiable

Fixed Assets

Predictable but with long timescales

This is the most uncertain value driver as it makes assumptions on growth in number of customers or increases in price at a given sales volume. Costs are generally internal and offer a higher degree of certainty than assumptions on increased revenue. Working capital is largely under the control of the organization and can therefore be predicted with a higher degree of certainty compared with revenue. Fixed asset values are generally known by the business. If the value is generated by a reduction (disposal) of fixed assets there is uncertainty on the re-sale value of the asset and the time it may take to dispose of the asset.

Introducing the C-ADAPT model  93

result in better profit margins for E-COMM. So, the list of functional areas directly or indirectly associated with the goal of improved profitability can include marketing and promotion (pricing, placement etc.); sales; operational excellence; supplier relationship management; competitors and other external factors. Let us list all As: marketing, sales, operations, suppliers, competitors. 3. D (Data discovery) The next stage of the model is ‘D’, which represents data discovery for the selected areas. At this stage the task is to list all the data sources and the types of data associated with the areas. When working on data, it is important to note that some data sets may be available within your organization and some data sets may exist outside the organization. Some data may be well defined and structured, but some may not be as well defined, or may be unstructured and scattered all over. Clearly, you may not have access to all the data sources directly (mainly outside data and some of the internal data), but at this stage, you should identify and list as many data sources as possible. Examples of outside data sets can include: conversations among people talking about your company; your products; your competitors or complementary products on social media platforms such as Twitter or Facebook. Most of the outside data will be unstructured data and you can create value and differentiation from that unstructured data. In most of the existing business intelligence tools, internal structured data analysis is already included and widely covered in many earlier books – in this book we are focusing mainly on the hidden potential of unstructured data and how many of the suggested technologies and tools in Chapter 4 can be used for unstructured data analysis. Some of this unstructured data can be collected using existing social media tracking tools easily available in the marketplace. Such data can also be collected using customer feedback processes online or offline.We can also classify these datasets as internal and external datasets. In our E-COMM example the following data can be attached to the selected areas from the previous stage: Marketing: from the marketing side, you can identify key information and data about customer profiles, market segment/demographics, buying patterns, buying triggers, buying processes etc. These data can be collected from existing sales data coming from your CRM (Customer Relationship Management). There will be data from responses to promotions you ran in the past and the responses collected from marketing and pre-sales data campaigns. In the case of digital or online promotions, these data can come from campaign management platforms such as Google Adwords, Google Analytics and Facebook and Bing Advertising. If it is related to offline promotions, then in most cases the marketing team manages internal data management systems

94  Introducing the C-ADAPT model

for manual updates etc. Social media conversations on portals like Facebook and Twitter can be collected using tracking tools like Hootsuite or Odimax. Data coming from social media will be mostly unstructured data and we will be using text analytics or sentiment analysis in creating value from this. Cost and pricing of the products sold will be available through admins of the web portal itself – if you are using Shopify, Magento or Wordpress then it is easy to access the admin section of the web portal and download product catalogue management data. Sales: when we work on improving the profitability of any company, sales become one of the most important sections of the project. It is often difficult to totally differentiate the data coming from sales and marketing, hence you may see some data sets or sources have an overlap between the sales and marketing sides. Things like most popular products in terms of number of items sold, and most profitable product based on gross margin, can be collected from CRM and SCM data. A least popular product sitting on the home or key landing page and not producing results can be rectified – such information can be found by combined analysis of CRM and Marketing data. Assuming that customers are looking at the products online, and then either calling at your sales office or chatting with your sales/support representative on the website to discuss the details of the product and buying them on phone – then this data will be available with sales and support MIS (management information system). Where assisted sales are taking place on E-COMM over the phone or on chat, then you need to have access to customer conversation with the sales representation and convert that into text (many software packages are easily available) if needed. This structured and mainly unstructured data will help you identify how many sales are coming from which channels and what the conversion rate of chat and phone conversations is. These MIS (or CRM) sources will also include their conversation data and can present insights such as what is being discussed, most popular discussion point, whether pain or information etc. Performance of sales and support team members can also be considered if the overall conversion rate is not in line with a few low performing team members – this data can come from the employee performance management system. Operations: it will be important to identify all customer touch points and see which are attached to sales and support and which are attached to operations. Operations are basically cost centres, hence the more smooth and optimized a process you can include in the organization the easier it will be for you to reduce costs associated with delivery.You can map the whole process from receiving a product or assembly parts in your warehouse to fulfilment and delivery of the products towards different orders. Identifying all the operations and accessing data related to resource requirement and production limit and outcome will help you in optimizing the processes where needed.

Introducing the C-ADAPT model  95

It may also help you identify those areas that are costing you more than others and it is an opportunity for you to analyze and identify those areas and make appropriate changes. You should identify how much resource is involved in the entire process and what the internal and external touch points are in the delivery process, including packaging and shipping. In order to improve operations, you can also consider analyzing system log files, encrypted emails etc., where semantic analysis will be able to help you identify key insights that may help you reduce costs by improving operations. Suppliers: your procurement system will give you access to all suppliers, their trading details, the goods/services they are providing along with terms and costing etc. This is another important area where, if possible, you may have opportunities to negotiate the terms and costs of goods with your existing suppliers in order to have direct incremental effect on your profitability by supplying the same products at the same price. List all the suppliers, their product pricing, margins on their goods, their delivery schedule etc. Sometimes it is worth using external data collected though the web, social media or other sources about the suppliers and what they supply – you may be able to find information about their relations and offers to others in your market – this information may give you some points for negotiating a better deal. It is also possible to collect information from other markets if you are willing to source products from overseas. In most cases product costs are low if you can manage to import product from low-cost countries such as China, Taiwan or India. Because E-COMM sells electronic products online, there is a high chance that you can source some of the products, if not all, from overseas – in that case data collected from other markets about suppliers should be considered in the analysis. Competitors: a study of competitors is always a very important factor in the profitability analysis of a company. If your prospective buyer is not buying from you, what (it may be a substitute), where else, how and why is he buying the product? There are many competitor analysis products available that can be used for businesses including e-commerce companies like E-COMM. Tools like SEMRush can help you get information about the digital marketing comparisons of the company and their competitors – how and what customers are searching for and how your competitors are responding is very important for any business. Competitor product range, their features, their pricing, their marketing and promotion activities etc. should be collected if not available using automated tools – as this information can be very meaningful in your analysis. Most of this data is external unstructured data – social media tracking tools and market research can be helpful in collecting this data and analyzing it using text and semantic analytics as discussed in Chapter 4. External factors: most of us are very much aware of PESTEL analysis (if not, please see Figure 5.2 and brief detail given below). It would be useful

96  Introducing the C-ADAPT model

FIGURE 5.2  PESTEL

analysis framework

Note: PESTEL Analysis:   A PESTEL analysis is a framework or tool used by marketers to analyze and monitor the macro-environmental (external marketing environment) factors that have an impact on an organization.

to perform PESTEL analysis as well for E-COMM because many external factors can also have direct impact on the profitability of the company. Data related to political, environmental, social, technical, environmental and legal factors will give you a lot of insights. For example, If E-COMM is importing some products from countries outside the European Union, say China or Taiwan, then your landing cost in the United Kingdom could be affected by political events such as Brexit. After the referendum resulting in a vote for Brexit the United Kingdom is preparing to leave the European Union and this has negatively impacted the value of the British Pound. This means you will get fewer dollars for the same amount of British pounds – leading to the issue that most of the export/import business outside the EU happens in USD (Dollars) and to buy any item from China or India, companies like E-COMM have to pay more than they were paying earlier. This change will have direct impact on the cost of products, leading to lower margins. Such political changes also affect the market drastically, and Brexit1 has led the United Kingdom into a lot of uncertainty that may go on to affect overall business in the market. People are now trying to spend less, and that is

Introducing the C-ADAPT model  97

affecting the topline revenue of businesses. One clear indication is that house prices in the UK have gone down following the UK’s vote for Brexit.2 Other external events and activities such as ‘Made in China versus Made in the UK’ product campaigns have always affected businesses based on their model. People who are selling British-made products have seen rises in revenue and others have seen a decline. What was the impact of the last Greenpeace campaign3 on your products made of plastic? This is relevant, as most of the products sold by E-COMM are using plastics. So, there are so many external factors and associated data available that you should consider including in your analysis. Most of the data related to such external activities come from web scraping, news portals or social media – where it comes in the form of raw text – and sentiment analysis along with popular themes or keywords such as Brexit, which can help you identify the trends and activities that may have some impact on your business in future. Let us summarize the list of Ds (data sets) you should include in this project: marketing and sales data from CRM and other sources (structured and unstructured data), Google Adwords, Google Analytics, Facebook Advertising, Facebook and Twitter feeds (unstructured data), product catalogue management data (structured data), sales and support MIS, call/chat conversation data (unstructured data), procurement data (structured data), open market supplies data, overseas supplies data, market research and digital competition data, external factors data (structured and unstructured data) etc. 4. A (Analysis and insights) In previous stages we have prepared a list of data sources that we should include in our analysis. The next stage is the ‘A’ that represents analysis and insights gained from data analysis. Before starting to work on the analysis part, you need to put all the data sources in the correct order and right format to ensure that they are all in sync and ready to be analyzed. Before real analysis starts, businesses perform a process called ETL on the data sets. ETL is short for extract, transform and load, three database functions that are combined into one tool to pull data out of one database and place it into another database. Extract is the process of reading data from a database. At this stage practising managers should take help of their technical team in order to establish data flow from multiple data sources as described in last stage and run ETL on that to make them analysis ready. As we have seen in the last stage, most of the data considered for this analysis is unstructured data, text analytics, semantics and sentiment analysis techniques, which will be used in different forms – but clean up, structuring and storage will take place before that (ETL).

98  Introducing the C-ADAPT model

Once data is cleaned and formatted, and processes are in place to perform ETL on real-time data as well, now you can start to work on the tools and techniques needed for the analysis. All the datasets should be analyzed now using multiple techniques discussed in previous chapters with the intention of using the collected insights for achieving the goal of improving the profitability of E-COMM. Many managers would not be feeling confident using complex analytical methods at this stage; in such cases they can seek assistance from the technology team in implementing analysis on the datasets. But even a non-technical manager can prepare a list of assumptions based on the data collected and ask their technical team to validate those assumptions in addition to finding other insightful information. If you have read all the chapters before reaching this page, you would have a reasonable understanding of the techniques you can use on different data sets for different purposes. Based on that knowledge you can talk to your team to help you produce the results to satisfy your assumptions.You as a manager can ask your technical team to find out the impact or pattern of sales during the Christmas season or when your competitor has released a new product – in effect you are providing them with a timeline and asking them to establish connections, patterns and correlations between different sales and marketing data set that is collected from different sources including CRM, marketing and customer feedback data stored in physical feedback cards.You can also get information about the price sensitivity of the market and impact of the last price change in the sales – this analysis can also be performed on sales and marketing data from different sources. When government implemented changes in taxes on imported products, how was our margin affected and did customers react negatively to a slight sales increase? In our presumptive example given above, when analyzing the data collected from the marketing sources some correlations can be established between the campaigns run by the marketing team, sales revenue received and sentiment of people around the product and the campaign run by the marketing team. Datasets received by the operation team can also be analyzed further to identify those areas of improvement and reduction in cost – in doing so, ultimately profitability can be improved for the company. At this stage, it is important for you to take help from your technical team, even if you are technically equipped and confident, so let us list out all the analysis expectations or assumption questions that you are planning to get answers to – here is a list of your As for E-COMM based on potential data sources: •

 hat are the key products that bring most margins and what triggers W their sales? Structured data analytics. What are the most popular and least popular products based on their sales volume, price points, gross margins and social media popularity? Structured and unstructured data analytics.

¡

•

¡

Introducing the C-ADAPT model  99

•

What are the best geographies for us based on rate of sales conversions? Structured and unstructured data analytics. Is there anything else we can include in our offerings – something like recommended products? Structured and unstructured data analytics. What kind of advertisements and what platforms give the best results in terms of sales and revenue? Structured and unstructured data analytics. How much money we are losing on deliveries and can we make some savings there? Structured data analytics. What is the best time (month/date/time) for us to run discounts or other promotional campaigns? Structured and unstructured data analytics. Who are the suppliers that give us the best service, quality products and lowest cost? Structured data analytics. How much time/resources do we spend in post-sales processes until delivery? Structured and unstructured data analytics. What are the common questions that customers ask on a phone call or online chat? Unstructured data analytics. How is the conversion rate and performance of individual sales/support staff? Structured data analytics. What would be the impact of government tax changes on our sales? Structured data analytics. ¡

•

¡

•

¡

•

¡

•

¡

•

¡

•

¡

•

¡

•

¡

•

¡

These are a few typical questions and associated analysis types – structured or unstructured – for a real business with practical questions. The list can be really big and the technology team will start to look for answers to these questions by performing different analysis on different data sets based up on the type of analysis and techniques described in Chapter 4 – eventually they will present the results back to the team in the best way possible. We will discuss this in the next section. 5. P (Presentation and visualization) The next stage in the suggested model is ‘P’, which represents presentation and visualization. This is the stage of the analysis where managers have to see the results and decide how they want to represent the information gathered and the insights identified and analyzed in previous stages. A picture speaks a

100  Introducing the C-ADAPT model

thousand words – sometimes this information appears in the form of visuals, infographics and dashboard; for managers or business leaders this can be very powerful. Sometimes findings needed to be supplied to other departments, so they can implement the changes suggested by the data analytics team and practising managers. Based upon the analysis needed and questions associated with them, it is important that appropriate ways of presentation are chosen. Here are some examples of E-COMM analysis presentations. When it comes to analyzing which sales channels are performing better than others, then a dynamic dashboard can be prepared for the team to understand the impact of campaigns and the results produced. Figure 5.3 shows where the performance of a web portal has been demonstrated on the basis of different channels and geographies:4 Individual channels performance can also be analyzed and represented in terms of identifying the best way to reach out to more prospects and convert them to customers – this method is based on ROI, where you can refine your marketing investment and campaigns. See Figure 5.4 for a sample report on the analysis of the Facebook activities of a company. Once we know which channels are performing better in bringing more sales to the company we can now try to represent the best season for the company from the sales revenue perspective. A bar chart or line graph plotted against a timeline with details of sales revenue can be a good way to represent the information. Figure 5.5 shows a sample chart.5 If we have access to a CRM system (like Salesforce) then analysis can also produce the right segmentations of the products and their sales numbers –

FIGURE 5.3 

Website metrics dashboard display

Introducing the C-ADAPT model

FIGURE 5.4

Facebook content performance dashboard display

FIGURE 5.5

Sales performance dashboard display

101

the example below shows the performance of different product offerings in different quarters and in different industry and geographies.6 A similar report for E-COMM can help you understand product-wise performance for different locations and segments (see Figure 5.6). Another sample report shown in Figure 5.7 presents multilevel segmentations for businesses to understand their customers better in addition to analyzing their product, market and channels. We have already discussed many different visualization tools, graphs and charts that can be used in data analytics and visualizations. Based on the situation and the individual case, appropriate charts, graphs or other presentation

102  Introducing the C-ADAPT model

FIGURE 5.6  Market

segmentation dashboard display

FIGURE 5.7  Multilevel

business segmentation dashboard display

tools can be selected by practising managers and implemented by technology teams. Comparative charts for competitor price, or a real-time dashboard of sales and campaigns can be a desirable choice for competitor comparisons but a simple performance text report might be appropriate for A/B testing results. Such analysis and presentations will then help different departments such as marketing, sales, supplier relations, customer relations etc. to suggest changes in increasing the profitability of the company. 6.T (Testing the value created, refine and repeat) The last stage of the C-ADAPT model is ‘T’, which represents testing the value created, refine and repeat. This is the last but most important stage of

Introducing the C-ADAPT model  103

this model. Based upon the findings of analytics and presentations by different departments, different strategies or changes will be introduced and their impact will be tested on a continuous basis. Some testing takes place while you are seeing the reports or dashboard prepared after analysis, because you can make real-time changes in the values and can see its impact on the other graphs and charts. Tools such as Tableau present a lot of options for managers to play with dynamic and real-time charts. Another testing example is when you make some changes on the basis of analysis and then you may have to wait for reactions or a response. Testing performed on a continuous basis and changes introduced into the business will help practising managers take further informed decisions that will help in creating more value for the business. This is a very important stage of the model, where testing is not just about the changes you have made and whether it is increasing profitability (that was the goal) but is also about creating value.You should take the investment made on Big Data preparation, resources and execution into account while working on ROIC (Return on Invested Capital) for checking value creation. As discussed in Chapter 3, if overall revenue (and profit) has been increased by our analysis and introduced changes, this is good news, but before jumping to any conclusion we have to also consider the investment you have made in the Big Data project including infrastructure and resources. Eventually you need not only to recover the cost of Big Data setup and execution, you also must create more profit than the company was making before the project. So, we can only say that this project has created true value when the increase in profitability is more than the previous profit and the total investment in the Big Data project and setup. If you have decided to run a marketing campaign, then you need to wait for some time and analyze the results before you can make a call on the next campaign. At this stage managers should prepare a list of expected results and then compare the actual results with them. As suggested at the beginning of this chapter, C-ADAPT is an iterative model where, based on the testing of the results, more changes can be introduced, or previous changes can be refined in order to produce better results. Post testing phase, if we realize that we have created value for the organization considering all the costs of Big Data, we can move to the next stage – challenges and goals – but in every iteration, based on the testing results, we move closer to the expected, refined or new goals – at every stage we need to compare financial value created by this analysis after reducing the cost of setting up and running the Big Data project. Let us use some fictitious numbers to check how we can calculate and decide if the E-COMM project has created value or not. We should always consider that the investment going into this project, including setup and execution and cost of implementation of changes and recommendations suggested as part of the project, should also be taken into account when we are calculating the return.

104  Introducing the C-ADAPT model

Let us assume that E-COMM has invested the following amount in starting and running the first iteration of this project. Table 5.1 shows that total investment in five years. We further assume that the initial setup cost will be a one-time cost and the project will be running with multiple iterations over five years and every year the cost of running the project will be increased by 10%. Hence the total investment in this project will look as follows (see Table 5.2). Table 5.2 shows that total investment in five years will be $14.2 million. Now we assume that with the help of this project and recommendation, the company will start to make more sales and eventually the profitability of the company will increase as well – that was the goal of the project. But we should not forget that the company will invest $14.2 million in setting up and running the Big Data project over five years and the project will only create value if the company makes at least $19.92 million (considering NPV as discussed in Chapter 3) in five years. Let us explain how we have reached $20.31 million. As mentioned in Table 5.2, from year 1 to year 5, the company will invest a total of $14.2 million on this project (addition of total row in Table 5.2). Assuming this investment will result in an increase in revenue from year 2 and from year 1 to year 5, business has received increased revenue of $0, $1 million, $2.5 million, $5 million and $10 million respectively, with an estimated 40% gross margin. That translates into an increase (or decrease) in the yearly operating income (cash flow) as $-4 million, $-1.2 million, $0.08 million, $2.34 million and $7.08 million from year 1 to year 5 respectively. If we assume 10% as cost of capital and calculate the NPV at 10% discount rate from year 1 to year 5 the NPV is $1.43 million. The goal with the investment was to increase the profitability of the business hence the initial investment of $14.2 million should generate at least $20.31 million ($14.2 million × 1.43) to increase profitability. On these assumptions the project should be given the go ahead. Here we are assuming that this is a high level analysis and it will further include a number of other headers. So overall profitability number may be higher – but the key take-away here is to identify the right way of evaluating the outcome of the project and identify whether executing this project will create value or destroy value. The C-ADAPT model is a very simple to use approach developed for practising managers with non-technical knowledge to identify and successfully TABLE 5.1  Cost of Big Data project (yearly for five years)

Big Data infrastructure/setup cost (year 1 only) Running costs IT/Data warehouse Internal 30 data analysts and staff

$2 million $1 million $1 million

TABLE 5.2  Example investment in Big Data project in five years

Heads

Year 1 Year 2 Year 3 Year 4 Year 5 Years 1–5 ($ million) ($ million) ($ million) ($ million) ($ million) ($ million) Total

Big Data infrastructure/ setup cost (year 1 only) Running costs IT/ Data warehouse Internal 30 data analysts and staff Total

2

0

0

0

0

 

1

1.1

1.21

1.33

1.46

 

1

1.1

1.21

1.33

1.46

 

4

2.2

2.42

2.66

2.92

 

Total investment in five years will be $14.2 million Increase in Revenue $m Gross Margin $m Big Data investment $m Increase in Op Income (cash) $m   NPV of Investment $m

0

1.67

4.17

8.33

16.67

 

1

2.5

5

10

4

2.2

2.42

2.66

2.92

14.2

−1.2

0.08

2.34

7.08

 

   

   

   

   

   

−4   $1.43

30.84  

Note: The goal was to increase profitability. With 10% Cost of Capital and an NPV of $1.43m the profitability (not only the profit) has increased. Hence Revenue should be more than $20.31 million.

execute Big Data analytics projects. We have also developed a worksheet that can be used by practising managers to implement the C-ADAPT model in executing their project successfully.

C-ADAPT worksheet As we have seen in the previous section, the C-ADAPT model is an easy way to structure and execute Big Data projects to create value for organizations. In every stage of this six-stage model, activities or tasks are associated with the outcome/results of the previous stage. Also, this is an iterative model, which means that when you are done with the last stage of the model – i.e.

106

Introducing the C-ADAPT model

testing the value created, refine and repeat – and you are satisfied that you have achieved or are moving close to achieving the goal you have identified, and in this process you have created positive value for the organization, you can restart the process from the first stage until you have achieved your goal fully or are satisfied with the outcome of true value creation. In this section, we present you with a practical tool of the C-ADAPT model. This is a worksheet that matches the six conceptual stages of the C-ADAPT model and in every stage subtasks and outcomes are listed in the worksheet. As you can see in Figure 5.8, the worksheet is basically a template guideline document or framework that can be used by practising managers to implement the C-ADAPT model for a Big Data analytics project. All parts of this worksheet can be mapped with the six parts defined in the model earlier. The first part of the worksheet provides space for practising managers to list all challenges or opportunities in their own words. It can be a one-line goal statement or a paragraph with more details. It is good practice to work with your team to do brainstorming to come up with the list of challenges and goals and then you can start to group them under different categories of challenges. This process can continue refining itself and several inputs can be considered before you come up with your final list of key challenges that needs to be addressed in this project. In our example project for E-COMM the key goal was to increase the profitability of the company – at this stage your worksheet will have one item in the first section (see Figure 5.9). Now when we have our goal listed on the worksheet, we will move to the next stage of the model where we will identify all the areas that may be

FIGURE 5.8

Typical C-ADAPT worksheet

Introducing the C-ADAPT model

FIGURE 5.9

107

Typical C-ADAPT worksheet (challenges completed)

related to the goal we are trying to achieve. From the previous section, we know that the key areas related to profitability for E-COMM are: Marketing, Sales, Operations, Suppliers, Competitor/Market etc. So now we would list these areas in the worksheet (see Figure 5.10). Here it is important to note that you need not to write areas as predefined business units or departments. You may observe that we have listed Marketing and Suppliers in the same list of areas where one is a process (marketing) and the other is an entity (suppliers) – the idea here is to prepare a list of areas that can give us a clue about the data sources and we can then move ahead to the next stage. Once you have set down the final challenges, you should list all the business areas/units/verticals that can have an impact on the challenges defined earlier in this process. Areas listed on your sheet can be partners, resources, activities, supplier relationships, employee relationships or anything else that you think can have an impact on the challenges listed above. We suggest you need not necessarily follow the predefined parts of the businesses only – you can think beyond suppliers, buyers, employees, market conditioning etc. Similar to the previous stage, the longer the list, the better, and then you can start to group them under a few common headers. In the next stage, we need to identify the data associated with the areas identified; in some cases these are internal data sets that can be found easily but in other cases they may be missing data sets or external data sets – once we identify them, we can make sure our data is ready for the analysis that comes next in the model. Please see how the E-COMM worksheet will look after we have finish the data discovery stage (Figure 5.11).

108

Introducing the C-ADAPT model

FIGURE 5.10

Typical C-ADAPT worksheet (areas completed)

The sheet has space for the data related to the areas identified in the previous stage. Identifying the dataset is a very important part of the project because you need to think in terms of what data is available already in the system, which can be downloaded in the company and what data is available outside the company. As suggested earlier, data can be internal or it can be external. Practising managers should be creative and proactive in terms of listing down what data sets can be collected and used in this analysis – reliance on the IT team can be a disadvantage at this stage because things like marketing data available outside the company may come from social media feeds or from product review portals and the IT team may not consider these datasets in their usual data analysis.We also suggest that practising managers should write as detailed information as possible about the datasets in this part of the worksheet. By doing that, managers can logically establish relationships between datasets without having to get into complex statistical models or understanding other mathematical calculations that many managers avoid.This is also the right place to connect a few data points on the basis of assumptions and put that forward to the technology team and ask if these assumptions are right. After the third stage our data sources are identified, the data collection and ETL process has started and data are now ready for analysis. At this stage, we need to list the analysis or assumptions/hypotheses, so the analysis can be performed in that direction. At this stage, identified datasets from the previous section and relationships established among them can be analyzed by different statistical methods as defined earlier.This is also an area in which some practising managers may not feel confident, hence we suggest that you should obtain

Introducing the C-ADAPT model

FIGURE 5.11

109

Typical C-ADAPT worksheet (data completed)

help from your technology analytics team who may have expertise in executing the statistical formulas and can present the results in the most appropriate, simplistic and easy to understand format. For non-technical practising managers, it may make sense to list the hypotheses or analysis expectations and the technology team can work on that accordingly. Your team will identify the analytics process based on the list of assumptions, hypotheses, challenges and the goals defined. Figure 5.12 shows how our E-COMM project will look. The next space on the worksheet is for the presentation – this is where you and your team will list the desired outcome, and what the outcome will look like. If the idea is to prepare a dashboard which will show a strategic positioning of the company to all the business leaders in the company then the presentation here should include the list of tools and techniques needed and what kind of presentation is needed. If the idea is to identify the key insights using Big Data analytics, implementing them by addressing different challenges, including the profitability of the company, then the result of the analysis will look like a detailed report about the findings and how to implement recommendations in order to increase the profitability of the company. In some cases, a presentation may look like a real-time analysis and report production on an ongoing basis. For example, if you want to know how positively or negatively people are talking about your business or brand on an ongoing basis on social media forums, then you need to have a system that keeps on analyzing new and upcoming data and producing the report for your appropriate action. Figure 5.13 shows the worksheet with the details given in the presentation section for the E-COMM project.

110

Introducing the C-ADAPT model

FIGURE 5.12

Typical C-ADAPT worksheet (analysis completed)

FIGURE 5.13

Typical C-ADAPT worksheet (presentation and decisions completed)

The last stage of the of the model is testing and repeat – the idea here is to list all the test scenarios on the basis of the analysis done and the associated changes introduced into the existing system. In some cases, where dynamic dashboards are implemented at the presentation stage, most of the in-system testing takes place by changing one or more variables and seeing the impact of this change on other variables. Where the report is presented in diverse ways and recommendations are given to be implemented as future strategy,

Introducing the C-ADAPT model

111

then at the testing phase you may need to wait until you receive the results of the implemented changes. Here we must also test if the project has created absolute value for the company. By having a robust testing process in place, you can measure the impact on every part of the business, hence can drive the changes for the performance improvement. Figure 5.14 shows the completed worksheet for our example company E-COMM where the main goal is defined at the top and and at the bottom you have the next stage of implementation recommendations and test/repeat details. As suggested earlier, the presented model is an iterative model and the process and analysis should be repeated to ensure better outcome and improved performance now and in the future. You can download a template version of C-ADAPT worksheet from www.creatingvalue.org.uk The E-COMM example that we have used here is a technical-heavy business because it is an e-commerce business for electronic products – there is a possibility that some managers may (not rightly) think that if their business is not very technical then they cannot easily take advantage of this unparalleled opportunity that Big Data is presenting to us. It is important to understand that all organizations are now collecting a lot of data and irrespective of your internal data set, your industry, market and other factors are anyway producing data that is beyond your control. Managers should also understand that not all the data needs to be recorded and analyzed – the more data there is, the better, but not all of it matters, hence working on a top-down approach where you define the goals or challenges first and then work further down is a logical approach. We

FIGURE 5.14

Typical C-ADAPT worksheet (testing filled)

112  Introducing the C-ADAPT model

have seen the complete flow of the C-ADAPT model from top to bottom and used C-ADAPT worksheet to make progress and create value from Big Data. Two key things to repeat here are: (1) The main component of this analysis is to include unstructured data in the mainstream analysis and produce results, and (2) The produced results should be measurable and should give positive ROI – meaning that the combined cost of running the project and the Big Data infrastructure, resources and execution etc. should be less than the net profit created, otherwise the whole exercise will be a loss making project in the long run. Let us take another example where the business is not as IT related as E-COMM was, to see that the C-ADAPT model and worksheet are equally effective and important for this example. Let’s assume you are Chief Operating Officer (COO) of a big and busy hotel chain – let’s name it BigDataHotels. BigDataHotels has more than 100 properties across the globe and because hotel accommodation is a competitive market, making decent profits in this business is a challenge. Unless you have optimum operations to keep the bottom line in control and a fabulous sales and marketing team who can attract many happy customers who will go away with a fantastic experience and will talk about your brand and promote you in different places including social media and other digital portals, profit-making in this industry is difficult. BigDataHotels has a historic presence offline and digitally they are as active and famous as they are offline. BigDataHotels receives a lot of data from multiple sources (structured and unstructured) and it is always innovating its practices to lead the industry in its own way. In the last six months, there has been a sudden dip in the bookings for one of the central London properties (in Bond Street, London) of BigDataHotels and no one could see any reason for this change. It cannot be a seasonal effect as other properties across the globe including in London are doing as expected, but this Bond Street Hotel is not delivering the expected results from last six months. Marketing, sales, housekeeping, all departments are working in the same manner as before and there is no outlying event or process that can justify the reduction in booking numbers or sales revenue. Now you as a COO and team are responsible for the task of identifying the root cause of the reduction in sales volume and revenue. As mentioned earlier that there are no signs of anything having changed in the last six months, so let us use the C-ADAPT model to handle this challenge, identify the root cause of the problem and bring the sales revenue back to its original level. We will be using the C-ADAPT model and worksheet to go through the entire process and then we will include a fully completed worksheet with the data from all sections of the C-ADAPT model.

Introducing the C-ADAPT model  113

C – Challenges and goals There are two key challenges mentioned in this example: 1. O  verall booking numbers and sales revenue have been declining over the last six months for the Bond Street, London property of BigDataHotels, whereas all other properties are doing as before and as expected. We must identify the root cause of this decline. 2. Once we have identified the root cause of declining sales of the Bond Street, London property then we must suggest how sales can be recovered and present a strategy for the same. A – Areas related Here are the key areas that affect top line revenue and sales volumes in the hotel industry; we can assume that we will find our reason by analyzing the data related to these areas. 1. S ales – we need to understand the processes followed, from marketing, sales and after-sales support provided to customers. All these sales-related data can be collected from the sales department. We also need to understand how many channels we have for sales and how they are performing, from partners and affiliates to digital marketing and high street travel agents. 2. Marketing – from segmentation to positioning, marketing plays a big role in bringing customers to the doors of BigDataHotels, hence marketing data will play an important role in this project. The marketing department can provide us with the data related to campaigns, offers and our routes to market that have been established from previous years, including digital and social marketing. 3. Customer service – the tour and travel industry’s revenue is greatly affected by the experiences of customers from the point they choose to book a hotel to the point they checkout and decide to share their stay and experience with their friends, family or with the wider world outside – it is therefore important that we include the customer journey and service data in our analysis for this project. We can get sales support and service data from the customer service department. 4. Operations – in most cases, operations affect the bottom line revenue of the business and it is ignored by analyses of top line or sales – but in some industries like travel, what happens after sales may have an impact on future sales. Especially in travel, smooth and optimal processes may lead to a happy customer who can eventually leave better feedback that

114  Introducing the C-ADAPT model

will play an important role in future sales. Hence, we will take operations data into consideration for this analysis. 5. Competition and market – sales revenue and volume are also a function of the competition’s performance. If your close competitor is offering heavy discounts or better-quality rooms, then you can expect to see some impact on your sales unless you have protected your position well or are competing penny to penny with your competitor. Apart from competitors such as other hotels, you should also consider substitutes like Airbnb that may have attracted some of your customers to their accommodation types. Market and competition research may provide a lot of meaningful information in this regard for this project. 6. Other open areas – all the areas listed above may have a direct relation with business performance and in most cases you will see data is residing within the organization. The political, economic and environmental situation can have an impact on business performance and you should also include these in your list of areas. D – Data discovery We have identified the list of areas that may be related to and impacting on the sales revenue for the Bond Street, London property of BigDataHotels – a challenge that we have listed in the first section of this model. Let us now prepare a list of data sets and sources we can use for these selected areas and can include in our analysis in the next sections. One more point is important to note: if the sales performance has been affected for the last six months then how much data should we consider for the analysis? As there may be some delay in seeing the real impact of any event it will be meaningful for us to include 12 months’ data from all the sources detailed below: 1. I nternal data: here is the list of data sets that we can gather from within the organization and which can be related to internal departments or events. This data is comparatively easy to access, and you may find your technical team is very familiar with and already using some internal data sets. a. CRM data (structured and unstructured data) – data coming from CRM will give you details about the customers, sales, volume, geography of the customer etc. – complete customer profiling data can be gathered from CRM. In addition to this, we can also have access to sales details, channels’ performance and the impact of marketing campaigns on the sales volume. b. Marketing campaign data (unstructured data) – it is important to understand what campaigns are working in the company’s favour and how much sales we are receiving in each campaign. Data from

Introducing the C-ADAPT model  115

advertising portals and agencies can help you get better insights. These days, customers are booking hotels online by using their laptop or mobile phones – this kind of data can be collected for the company by using web tracking tools such as Google Analytics and Google Adwords. c. Customer care data (structured and unstructured data) – in most hotels there exist clear processes for customer support and services where conversations with customers are recorded and stored in the database.You can have access to this information, such as number of calls, typical complaints, issue resolution reports etc. and interpret this to understand if there are any issues in the hotel and whether service staff are performing as expected.You can also have access to the customer feedback provided on a regular basis and stored in the system. d. Supplier and procurement data (structured and unstructured data) – this data will be available in the ERP systems of the company where a big part of this data is directly associated with operational processes and relationships with the suppliers of different products and services in the company.We can know what the process of welcoming a guest is and what items are provided in their room – where do these items come from and how much are we are paying for these? How long have these suppliers been working with us, etc. – this information can be collected from the procurement data. 2. E  xternal data: there are many data sources that exist outside the organization and produce relevant data on the business and company. Some of these data are curated and stored by some departments of the company and others need to be fetched and stored for our project. Here is the list of external data sets that we should consider in the analysis: a. Social media data (unstructured data) – it is possible that BigDataHotels may already be tracking and storing social media and digital marketing data using tools like Hootsuite or Odimax, but if it is not available, then we need to collect the required data from social media sources and store it within our organization to perform analysis on that. We can track the brand mentions about BigDataHotels, its competitors and associated keywords. We can get access to the data about what customers are saying about our services on portals like Facebook and Tripadvisor after checking out from the hotel. In most cases such feedback is more transparent and valuable than the feedback collected by staff members within the company, because on social media or other portals like Tripadvisor, customers are free to write about their experience without having

116  Introducing the C-ADAPT model

any obligation to the staff member who was asking for feedback at their checkout desk. b. Market Research (structured and unstructured data) – with the help of independent market research reports, and details about your competitors and complementors working in your market, you can understand the business impact from outsiders’ perspectives. Social media data collected for competitors and others in the market can also add value in this analysis. People’s feedback about your competitors can be compared with the feedback about your service for further insights. c. PESTEL data (structured and unstructured data) – we have discussed PESTEL analysis in the previous section. It will be important that we access the data, details and information about political, economic, social, technological and legal changes and their impact on the business.This data can also be collected by social media tracking tools, open market reports and other sources. A – Analysis and insights Now we have listed all the areas related to the challenge and goal we have for this project and identified all the data sources we may need from internal and external sources; we should now prepare for the analysis stage. At this stage, you may need to work with your technology team to perform ETL operations and clean and store all structured, semi-structured and unstructured data in a form that is ready for the analysis. Once all the setup is done and the data is ready to be used and analyzed, we must identify the kind of analysis we are going to perform and what questions that analysis will answer. As we did in our last example (E-COMM), if you are not technically comfortable then you can come up with the list of questions that you want answers for and your technical team can setup and run the analysis and present it in an appropriate form. Here are the key questions to which we need answers from this analysis: 1) How are all sales channels performing? 2) How are the different segments (geography, source, medium etc.) performing with respect to each other? 3)  How was the performance of all the marketing campaigns in the last 12 months? 4) What is the customer feedback collection process and how was feedback in the last 12 months? 5) Who are all our suppliers, how long have they been working with us and how is their performance?

Introducing the C-ADAPT model  117

6) What is the customer feedback from outside sources, e.g. social media and Tripadvisor etc.? 7) Are our competitors taking our sales, how and why are customers reaching out to them instead of us? To find answers to these questions your technical team can use multiple techniques and tools as suggested in Chapter 4. There may be several setups and tools needed and once you have the report or analysis presented to you, hopefully you will be able to see the insights that can help you face the challenge we defined in the first section of this project. Here are some of the suggested analyses that could be performed on the data sets to identify the key insights and their inter-dependency: 1)  Market and competitor comparison analysis using all curated data including social media data. 2) Sales analysis. 3) Segmentation analysis. 4) Marketing analysis. 5) Customer feedback curation and analysis – from internal and external sources using sentiment analytics. 6) Supplier performance and activity analysis. May other analyses could be performed based on iterative findings and decisions. In the next section, we will discuss how the result of the analysis will be presented to make the best of this process. P – Presentation and visualization After running multiple analyses, as mentioned in the previous section, on the data that is collected from internal and external sources the results will be presented in different formats based on their usage. 1) Sales report – the sales report can be presented as a dynamic dashboard where you as COO can make changes in the dates and other factors to see the impact on sales volume and revenue. Let’s assume you have observed that sales revenue and sales numbers have been declining in the last six months as we know but this is not a sudden dip – it is a gradual decline that was started about nine months previously and only became visible from the last six months. This means there was no one sudden incident that caused the decline but that something was happening in the previous months and its effect started to be visible from the last six months.

118  Introducing the C-ADAPT model

Next you see that sales numbers coming from online channels like digital platforms, social media and Tripadvisor etc. are affected more than those sales which come from offline sources. So, we have two clear findings here: a)  Sales numbers have been declining in a gradual fashion, and b)  Online sales are more affected than offline sales. 2) Segmentation report – sales segmentation report has suggested that no specific geography or location has shown any big change in the sales numbers. All the places are showing fewer sales. So, we can conclude that: c) Sales decline is not related to any geography or another demographic divide. 3) Market and competitor comparison report – this report will be the part of dashboard where you can change the dates and other factors to see the impact on competitors’ performance. It was not possible for you to get the sales data from competitors’ companies but their online popularity and comparisons on key word performance can be accessed from Google Adwords and Analytics-type tools. Frequency and rate of conversion from that data is not giving any clue as to why that one particular competitor is taking advantage of the declining sales numbers of BigDataHotels. The finding here is: d)  Loss of sales for BigDataHotels is spread across all the competitors. 4) Customer feedback report – this report presents an aggregated view of customer feedback and customer service data in the last 12 months. Date, time and source of feedback can be changed dynamically to see if there are any patterns established in customer feedback. It was realized that there is a huge volume of negative feedback coming from online sources and social media but there is not much difference in offline, on-premises feedback. Filtering further for only online feedback, we could see that increase in this negative feedback trend started around 10–11 months ago. Now we can further analyze the content of the negative feedback to see if all the customers are complaining about the same or similar issues. On text analysis we spotted that a number of customers are complaining about neck and back pain caused by the pillows provided in the room – this was a good find, but we have to see more details. Key findings from feedback so far are: e) Customers are more likely to give online feedback for negative experience than offline, on-premises in front of our staff members. f) People are more likely to leave negative feedback online than positive feedback. g) There is a surge of online feedback from all platforms and many of them are complaints about neck and back pain.

Introducing the C-ADAPT model  119

5) Customer service report – this report presents the details about customers contacting staff members before, during and after their stay at BigDataHotels, Bond Street, London.When we separate the number of service calls made from pre-sales, post-sales and during a stay, we realized that the number of calls during a stay are increased from the last 11 months. Further text analysis was performed on the recorded call-scripts of hotel service calls; it was spotted that many guests complained about the quality of pillows and/or mattress and some of them requested a change as well. Key findings from this report were: h) Many guests reported about the quality of pillows and mattresses during their stay. 6) Supplier report – the next report to look at was the supplier report, and what items they are supplying to the hotel. It was easy to spot that a new supplier was appointed about 11 months ago and they have replaced most of the pillows and mattresses in the property. Now things were starting to make sense, but still it was unclear why this information had not been reported in the last 12 months. That would need some investigation – and we will discuss that a little later – but here is the finding from this report: i) A new supplier started to supply new pillows and mattresses that caused pain and discomfort to many guests. So collectively here are our findings that we have extracted from the analysis: a) Sales numbers have been declining in a gradual fashion, and b) Online sales are more affected than that of offline sales. c) Sales decline is not related to any geography or another demographic divide. d) Loss of sales for BigDataHotels is spread across all competitors. e) Customers are more likely to give online feedback for negative experience than offline, on-premises in front of our staff members. f) People are more likely to leave negative feedback online than positive feedback. g) There is a surge of online feedback from all platforms and many of them are complaints about neck and back pain. h) Many guests have reported about the quality of pillows and mattresses during their stay. i) A new supplier started to supply new pillows and mattresses that caused pain and discomfort to many guests. Based on these findings here is the logical conclusion that we can draw – around 11 months back a new supplier started to supply new pillows and mattresses and those have caused neck and back pain in guests. Many guests have complained about the same and requested a change of pillow etc. Many

120  Introducing the C-ADAPT model

such guests have not left feedback while checking out, presumably thinking that giving negative feedback to a staff member in person was not a comfortable experience. Hence, many of them left negative feedback online and on social media. In the last 12 months, this negative feedback has accumulated online and started to show on online sales portals and partners like Tripadvisor’s portal. As most online and mobile users rely a lot on online feedback, this negative feedback has affected online sales numbers in a big way and eventually sales numbers have started to go down. So, we have sort of spotted the issue causing declining sales but still we have few unanswered questions and another challenge to address: Why was this issue never spotted? Most guests use their pillow and mattress while they sleep at night and then they call and report the issue over the phone and the issues are rectified. These calls were recorded but daily reports produced only the number of calls and not the script of the conversation. There was a clear indication that there are many calls received during night hours but because no one has converted the recorded conversation to text and run analytics on that unstructured data, the issue was never identified and resolved. In our analysis we have identified some dots and in the process of connecting the dots, we have worked out what the conversations were about and eventually we have reached the real reason for the declining sales. Decision - we have now identified the root cause of the problem that BigDataHotels was facing and appropriate actions can be taken in order to replace those products from the supplier that were causing pain to the guests. The next goal was to restore sales to their original level – once the supply issue is fixed, then this project should be assigned to the sales and marketing team so they can run campaigns relating to these changes and influence people to start using the hotel as before. Online reputation management projects should be initiated to recover the positioning of the brand and improve it further. T – Testing the value created, refine and repeat As a decision has been taken in the last stage of the C-ADAPT model, managers should be continuously tracking and testing online sales performance and keeping an eye on new feedback etc. The dynamic dashboard for sales, marketing, supplier and customer service report will help the team to keep the top line revenue up, but all the time keep customers happy. More importantly, this project is not just about identifying the key issue or recovering sales – it is about value creation from the analysis we have performed. As discussed in Chapter 3, overall value created should be calculated

Introducing the C-ADAPT model  121

FIGURE 5.15 

Typical completed C-ADAPT worksheet

on the basis of value created and the investment made to run Big Data projects – if the difference comes out as positive, then that means the project has created value and it should be further enhanced and repeated. Here is the completed C-ADAPT worksheet (see Figure 5.15) that reflects the whole project – one section should be filled in at a time but for this project we are presenting the final sheet – it has been filled in as we have followed the C-ADAPT model and discussed multiple issues while working on the project:

Summary This was the most important chapter and the key takeaway from this book. After learning about Big Data, value creation concepts and the tools and techniques, what is most important for the practising manager is the systematic model and framework used in implementing these learnings in practice. Whether you are a new or an experienced manager who has led Big Data projects in the past, you will find immense value in the C-ADAPT model and associated worksheet. As you have seen in this chapter, the C-ADAPT model and worksheet are two powerful tools that give you the insight and approach for any Big Data project for your business. We have considered two different cases and followed the processes here so that you can correlate this with your own projects and lead projects with confidence. There are two things we should always consider while working on the C-ADAPT model:

122  Introducing the C-ADAPT model

• •

 ig Data analytics should include unstructured analytics, otherwise your B project will not be much different from traditional Business Intelligence (BI) projects. Practising managers should be able to measure the overall value created from the project and consider the cost of Big Data project setup, infrastructure and resources etc. If your value created is not greater than that of the investment made, then irrespective of how much revenue you will make after this project or costs you will save, the overall created value will be negative and the project cannot go further.

If there are two key takeaways from this book, we would suggest they are the C-ADAPT model and the C-ADAPT worksheet.

Notes 1 Brexit is a term that has become used as a shorthand way of saying the UK leaving the EU – merging the words Britain and exit to get Brexit, in the same way as a possible Greek exit from the euro was dubbed Grexit in the past. 2 www.independent.co.uk/news/business/news/brexit-latest-house-market-buyslooking-for-new-homes-falls-lowest-level-3-years-a7199516.html 3 Greenpeace: Greenpeace is a non-governmental environmental organization with offices in over 40 countries and with an international coordinating body in Amsterdam, the Netherlands. Founded by Canadian and US ex-pat environmental activists in 1971, Greenpeace states its goal is to ‘ensure the ability of the  Earth  to nurture life in all its diversity’ and focuses its campaigning on worldwide issues such as climate change, deforestation, overfishing, commercial whaling, genetic engineering and anti-nuclear issues. 4 https://10az.online.tableau.com/#/site/demodepot/views/Marketing-WebsiteMetrics/SiteUsageDashboard?:iid=1 5 https://10az.online.tableau.com/#/site/demodepot/views/Sales-ExecutiveSalesforce/SalesGrowth/[email protected]/Test21?:iid=1 6 https://10az.online.tableau.com/#/site/demodepot/views/Sales-ExecutiveSalesforce/Segmentation/[email protected]/Test21?:iid=1

6 BIG DATA CASE STUDIES

In previous chapters, we have seen the importance of Big Data and value creation and how you can use Big Data to create financial and non-financial value for your organization. We have also noted that some of the early adopters and leaders in this space have already started to use Big Data to create value. This chapter is about those leaders, early adopters and their successes and experiences. In this chapter, we have included a range of case studies around Big Data analytics and how organizations are using this latest technological invention for value creation. To ensure that we cover a wide variety of cases, and you as a reader get a holistic experience of Big Data analytics and its usage, we have covered cases from growing technology companies such as Ooredoo (Qtel) to not so popular but very important and valuable agriculture-related companies such as John Deere. It is very easy to see that should you have any problem statement as will be seen in the case studies to follow, you would be able to reach an impressive solution using the suggested C-ADAPT model. We can see that most of the stages of C-ADAPT can be applicable directly in most of the cases but as we do not have enough detail about the real investment and return numbers, we are not able to do the final ROIC calculations, as illustrated in Chapter 3, about the net value created. However, in such cases we can safely assume that as the organizations have successfully achieved the goals they were aiming for, their overall gain was more than the investments, maintenance and resource requirement etc. that would eventually lead to a positive value-created. The authors of this book have worked first hand on some of the cases given below, either as part of their previous companies or as consulting assignments. We have also included some cases that have been contributed by other practising managers in different companies and industries. The idea here is to present cases where you can establish your learning from previous chapters and correlate it with the challenges and goals of your organization

124  Big Data case studies

to see how you can run such projects to create sustainable value in the organization.

Ooredoo (formerly Qtel) Rebranding and message amplification Background Qatar Telecom (Qtel) is a leading telecoms provider in the Middle East, and ‘the fastest growing telecom company in the world’. Ooredoo provides mobile, wireless, wireline and content services with market share in domestic and international telecommunication markets, and in business (corporations and individuals) and residential markets. It is one of the world’s largest mobile  telecommunications companies, with over 114 million customers worldwide as of September 2015. Ooredoo has a market capitalization of QAR 301.2 billion as of September 2015 and was named ‘Best Mobile Operator of the Year’ at the World Communication Awards 2013. Ooredoo is a leading international communications company delivering mobile, fixed, broadband internet and corporate managed services tailored to the needs of consumers and businesses across markets in the Middle East, North Africa and Southeast Asia. Ooredoo has a presence in markets such as Qatar, Kuwait, Oman, Algeria,Tunisia, Iraq, Palestine, the Maldives, Myanmar and Indonesia. The company reported revenues of US$9.1 billion in 2014 and had a consolidated global customer base of more than 107 million people as of 31 December 2014. Ooredoo’s shares are listed on the Qatar Exchange and the Abu Dhabi Securities Exchange. Opportunities/challenges Qtel were undertaking a global re-branding exercise, announced at the Mobile World Congress 2013, to unite all their country-specific brands under the new name: Ooredoo. In Arabic, the word ‘Ooredoo’ means ‘I want’. New branding was coming with a global yet personal touch to please their existing and future customers and other stakeholders. The Middle East operator had enlisted the help of a global public relations and advisory firm, FTI Consulting London, to guide them through the successful development and implementation of the new branding strategy. The telecom giant wanted their new brand messages to reach a wider, targeted audience to make a significant, lasting impact. In the close-knit communities of the Middle East, the majority of online conversations are conducted either in private or among users who know each other offline. Qtel were looking for effective ways to get their messages out

Big Data case studies  125

to the people who needed to hear them, particularly during the rebrand to Ooredoo. Method The biggest task for FTI was to find a fast and efficient way to penetrate this new branding message in both a majority non-English-speaking domestic audience and an English-speaking global audience at the same time. Broadly viewed, it was a marketing and branding problem and the first task was to identify the areas of the communities where the new branding message should be pushed with an impactful campaign. FTI started to work with a technology startup team based in London, who were working in the field of social media and marketing analysis. They implemented SNA (Social Media Analysis) algorithms to identify the most popular and related topic among their target market and then used that data to identify the most popular nodes (social media accounts) working on or talking about these topics. FTI worked to identify local and global digital influencers across multiple channels who could be engaged to amplify the spread of Qtel communications. A Social Media Analytics system and Smart Text Analysis were implemented and used to identify key influential media, networks and individuals. The whole PR and marketing strategy was planned and executed on the basis of insights identified by the FTI team.An analysis of content and conversations established who the key nodes in social networks were, their favourite topics and channels and how they might be engaged as brand advocates and message amplifiers. Each identified influencer was further analyzed and categorized to understand their type and level of influence, and the audiences they engaged with. Internal as well as external marketing and communication data were used to establish the pattern of influencers’ communications and their areas of interest. Also, in the World Mobile Congress (WMC) 2013 the whole overnight rebranding exercise took place where on day one it was all about Qtel and the overall surroundings of the event were themed blue – the next day everything was converted to the new brand and the whole environment was decorated in with vibrant red – the new brand colour.1 At the same time new messages were reaching the identified influencers and subsequent networks. Jack Morton, a global branding agency was hired by Qtel to execute the overnight transformation. Results Having identified key influencers on numerous digital channels, and for different topics, FTI and Qtel directly reached these influencers and were able

126  Big Data case studies

to engage these powerful social networkers as brand advocates and message amplifiers. As a result, the telecom company’s brand communications reached a wider, more engaged audience across a larger area of the MENA region, giving the new Ooredoo branding an impactful launch. With the Social Media Management and Analytics system in place Qtel could continue to identify and engage with influential online users to ensure the ongoing amplification of their news and marketing content.

Domino’s Pizza Understanding buying behavior Background Domino’s Pizza Inc. is an American pizza restaurant chain and international franchise pizza delivery corporation headquartered at the Domino Farms Office Park in  Ann Arbor Charter Township, Michigan, United States. Founded in the United States in 1960, Domino’s operates in 81 countries, making it the second-largest franchised pizza chain after Pizza Hut. Domino’s Pizza is recognized as the world’s leading pizza delivery company. With a global customer base of digitally engaged consumers, Domino’s are keen to make excellent use of wider digital data to optimize their operations. Opportunities Dominos Pizza is known for developing its processes and technologically inventing new ways of growing the business. By pushing their limits and using new technical ways of reaching customers, it is now possible for a customer to place an order for a Domino’s pizza from Twitter, Facebook, Smart Watch,TV or even from a car entertainment system. Keeping the order button available everywhere for customers is known as ‘Domino’s Anyware’. Domino’s Anyware has started to capture customer data from many directions and it has presented a huge opportunity for the company to analyze and profile their customers on the basis of their buying behaviour and motivate them to place more and frequent orders and serve customers better than before. Method The Domino’s team has started to capture data from all available channels such as text messages, Twitter, Pebble, Android, Amazon Echo etc. and then passes all this data to the information management system. This data is

Big Data case studies  127

then combined with other data sources such as location, demographics and postal data etc. This new enriched data helps Domino’s to build a unified customer segment and profiles. This new data presents meaningful insights for all departments – from Operations through to Sales. Information collected through the group’s point of sales systems and enrichment data adds up to 85,000 data sources, both structured and unstructured, pouring into the system every day. A team of data analytics consultants has worked on the data and come up with hidden insights, identifying the areas for improvement and associated data stream with the selected areas of Dominos’ business. Results After implementing this Big Data analysis, Domino’s have the ability to not only look at a consumer as an individual and assess their buying patterns, but also identify the influencers in the buying process. Domino’s have access to data about customers’ locations, their buying pattern, their usage of discount coupons etc. Now they know about which channels are performing better and how to improve sales on different routes. Domino’s is now working to build tailored coupons and product offers. Now Domino’s group receives 55% of orders from online or digital routes and that presents huge benefits to them over the competition.2 Domino’s Pizza is transforming to an e-commerce portal for very specific food items with all these innovations. Now they have moved away from a time when they were just processing data on sales and operational metrics in their warehouse, every business unit under the Domino’s umbrella is looking to leverage data to make them faster and more cost effective.

Leading antivirus company Increasing engagement and driving revenue Background One of the US’s fast-growing internet and mobile security brands was looking to significantly ramp-up marketing and communication management processes, with emphasis on increasing awareness, engagement and brand equity in order to positively impact their top line growth. The antivirus software market is highly congested, with the top 10 vendors holding approximately 88% of market share. This antivirus company has been able to establish a reasonable customer base, but now faces the challenge of continuing to build on that initial success.

128  Big Data case studies

Opportunity/challenges Social media was identified as having a vital role to play in the company’s marketing efforts to overcome these challenges. With limited resources, the client needed to drive more traffic to their website and product downloads, increase new sales, retain existing clients and increase revenue. A data analytics firm was engaged to help them to analyze and use social media to achieve these goals. Method A data analytics firm conducted a deep analysis of the client’s social media activities and the activities of the competition. A Smart Text Classification engine was used to implement a thematic approach to identify and analyze the key themes in the online conversations of the company, competition and consumers. This approach allowed them to identify: ----

 ontent that excites and engages online users, generating likes, retweets, C comments, etc. Gaps in coverage between customer conversations and brand content How messages and content vary across demographics such as location, gender and time

A Social Media Analytics System was used to conduct sentiment analysis to ascertain which themes and topics inspired positive and negative feedback. The system was also used to identify key influencers, whose brand advocacy could have a tremendous impact on the overall brand outreach. These analyses were combined with sales and marketing data from Google Analytics to give a comprehensive picture of the role of social media in driving web traffic and, ultimately, sales. As a result of the in-depth analysis, our experts were able to make many valuable recommendations to tc Client as to how they should use their social media assets to increase website traffic and sales. Results The client implemented the recommendations made as a result of the data analysis. Using a bespoke management and monitoring system they were able to increase their presence on social media, track the impact of their campaigns, find new customers and track the flow of customers from first interaction to purchase, on an ongoing basis.The analytics system enabled the client’s marketing team to accurately create and target their digital marketing

Big Data case studies  129

activities to engage existing and new customers, resulting in reduced marketing spend, increased conversions and improved revenues.

Gate Gourmet Use of data for competitive analysis Background Gate Gourmet, whose parent company Gategroup is headquartered in Zurich, Switzerland, with corporate centres in Reston,VA, and London, UK, prepares 250+ million airline meals a year at more than 120 flight kitchens around the world.The company conducts as many as 1,200 food safety checks each day and maintains a food safety record that’s considered among the best in the foodservice industry. Opportunities/challenges A team of consultants, working on behalf of Gate Gourmet, were looking to establish the current market position and identify new opportunities for this major independent catering provider for airlines and rail companies. Gate Gourmet were looking for new avenues to exploit to further develop and grow their brand; to conduct competitor analysis; and to develop a deeper understanding of their market position. Method Using the Data Analytics System, the team conducted a comprehensive assessment of the marketplace, comparing the client’s online activity, presence and brand equity with that of other key industry players. Online mentions of the brand and their competitors were tracked and analyzed to identify mediums which were performing well or – more importantly – were not yet being exploited, including social media channels, news websites, forums and blogs. It was found that while Gate Gourmet were receiving the most brand mentions on social media, their competitors took the bulk of the traffic for larger news stories and articles. These mentions and wider conversations were analyzed to identify recurring themes, highlighting potential new avenues for the company to exploit. Analysis of popular content from across the industry was conducted to shape future content creation and optimization to maximize the chances of content being engaged with and shared.

130  Big Data case studies

The Data Analytics System was used to identify industry influencers and advocates, both for the brand and their competitors.These influencers can be targeted to enhance the spread of campaigns and company messages. Analysis was conducted to identify the sources of content and it was found that the vast majority of posts were made via the web, i.e. from laptop or desktop computers. To increase social sharing, content should be optimized for viewing on mobile web devices, prioritizing optimization on the devices identified by the Data Analytics system as being used by the target audience. Key word analysis found that people were discussing jobs and careers at Gate Gourmet, using social sites and the web. This highlighted a potential opportunity for recruitment drives to be made via social channels to find new pools of talent. Value/results The analysis revealed that there were several online channels which were being underexploited, highlighting opportunities for greater engagement, online activity and internet coverage. Avenues were identified where optimized content can be shared and distributed to maximize engagement and establish the company as the industry leader on all fronts. Content should be optimized for viewing on both web and mobile web devices to maximize interactions and sharing. Based on the analysis of industry influencers and the wider audience, recommendations on style, topic and mediums for content were made to increase engagement and sharing of content and boost the brand’s share of heart and brand equity in order to ultimately increase the overall brand health of Gate Gourmet. Analysis of conversations around the brand and the wider industry uncovered extensive interest in jobs with the company, so a recommendation was made to utilize social and other online media to recruit engaged and enthusiastic talent.

Tesco Knowing customer insights from data Background Tesco is one of the largest retailers in the world and is originally from the United Kingdom, founded in 1919. Currently they have stores in 12 countries under different brand names. In the fiscal year (2013/14) they had $110 billion in group sales and $3.6 billion in group profit before tax. Over 500,000 employees work in 7,599 stores around the world, including franchises.

Big Data case studies  131

Their largest market is the United Kingdom, with almost 3,500 stores and over 310,000 employees. Next to supermarkets, they also operate, among others, petrol stations, a bank and mobile phone, home phone and broadband businesses. Of course they also have a loyalty programme and together these business sections create massive amounts of data. They started using that data in the 1990s and have ever since expanded. Opportunity Tesco was one of the first retailers to move into data collections from their customers. Tesco Clubcard is a loyalty card introduced by Tesco in 1995 – and they have been collecting data about customer spending and behaviour since then. Apart from using Clubcard data from sales promotions and discount codes, there was a huge opportunity for Tesco to understand customers’ behaviour in the context of other data and factors and use that insight to optimize the processes, and improve the bottom line as well as the top line. Method Two thirds of Tesco customers use Clubcard now and leave data about their shopping activities with Tesco. This customer spending and buying behavior data is analyzed with many other data sets and factors, in order to bring about improvements in different parts of the business. Tesco data analytics has considered different internal data sets such as customer profiling and electricity usage of refrigerators and external data sets such as weather reports and buying patterns in changing/rainy weather. This kind of analysis has empowered Tesco to predict the demand and sales of their stores and save food wastage. Result With the help of weather reports and outside data, Tesco is able to predict demand for most of its products.This also helps in reducing the food wastage as well as saving huge amounts of money. By combining weather data and sales data they know what to expect and in the past years that has resulted in over $9 million less food wastage in the summer, $47 million less wastage due to optimized store operations and $78 million less stock in warehouses. Another way Tesco benefits is by optimizing the temperature of refigerators in the store and therefore saving electricity costs. Tesco has carried out this analysis project with the IBM research laboratory and by optimizing the refrigerators based on analytics results, Tesco has reduced their energy bill by almost $25 million a year. 

132  Big Data case studies

Hence Tesco’s use of data analytics has not only helped them increase their sales revenue by understanding their customers better, it has also helped save money by reducing food wastage and electricity bills.

Delta Airlines Data is helping them find lost baggage Background Delta Air Lines, Inc. is a major American airline, with its headquarters and largest hub at Hartsfield-Jackson Atlanta International Airport in Atlanta, Georgia. Delta Air Lines serves nearly 180 million customers each year. In 2016, Delta was named on of Fortune’s top 50 Most Admired Companies in addition to being named the most admired airline for the fifth time in six years. Additionally, Delta has ranked No.1 in the Business Travel News Annual Airline survey for an unprecedented five consecutive years. Challenge The airline industry is an incredibly difficult space to operate in, with intense competition and razor-thin margins. Despite the many challenges airlines face, Delta has emerged as an industry leader over the last several years with superior financial and operating performance. All airlines know a major concern for passengers is lost baggage, particularly when they are on a flight that’s delayed and there are missed connections involved. Delta looked further into their data and created a solution that would remove the uncertainty of where a passenger’s bag might be. Delta is known for its innovative and customer-centric approach. They have decided to solve one of the most common problems passengers face – tracking and finding lost baggage. According to Paul Skrbec, a spokesman for Delta Air Lines, ‘every customer has had the experience of boarding a plane after checking their bag and wondering if it was there.’  Method Delta has deployed Radio Frequency Identification (RFID) baggage tracking technology, a first for U.S. carriers, providing customers with improved realtime tracking of luggage throughout the travel experience. This move marks a historic shift for Delta and the 120 million bags it handles annually.  Delta has invested over $100 million in airport baggage systems to track and improve its baggage handling. They are using advance tools for baggage data collection and analysis, and now Delta’s operations teams at airports

Big Data case studies  133

and headquarters have been able to better identify key causes and trends in mishandled bags and implement effective solutions. By integrating real-time flight data into its baggage systems, Delta now automatically alerts baggage handlers when connecting bags need to be transferred directly to another plane instead of being sent through the airport’s luggage sorting system. Delta has also begun sharing baggage tracking data with travellers, who can now follow the progress of their bags minute by minute on the Delta app and website.  Customers can now snap a photo of their baggage tag using the ‘Track My Bag’ feature on the Delta app and then keep tabs on their luggage as it makes its way to the final destination. Even if a bag doesn’t make it on the intended flight, passengers save time tracking it down. Finding a new way to put Big Data to use for the benefit of their passengers put Delta out front in a competitive market. Results By bringing data to what was previously a highly manual operation, Delta has reduced its mishandled baggage rate by 71% since 2007 (more than any other airline) and improved its customer experience. Delta is now a leading U.S.based global airline for baggage performance. In 2015 Delta led U.S. global airlines in DOT bag performance while setting six monthly DOT records and a full year record.3 Delta has made use of more advanced data in its efforts to better engage customers and generate loyalty. The airline combines customer data from flight purchases, routes flown and credit card spending to piece together customers’ demographic profile, travel habits, spending ability and even what company they work for. The company uses that data to carefully tailor promotions and target customers with whom the airline senses there is more opportunity.

Intel Saving manufacturing costs with Big Data Background Intel Corporation  is the world’s largest semiconductor chip manufacturer based out of the USA. Founded in 1968, Intel is the inventor of the x86 series of microprocessors, the processors found in most personal computers (PCs). Intel supplies processors for computer system manufacturers such as Apple, Lenovo (formerly IBM),  Hewlett Packard  and  Dell. Intel also manufactures  motherboard  chipsets,  network interface controllers  and  integrated

134  Big Data case studies

circuits,  flash memory,  graphics chips,  embedded processors  and other devices related to communications and computing. Intel has eleven fabrication facilities and six assembly and test facilities around the world. It has changed the global marketplace dramatically since it was founded in 1968; the company invented the microprocessor, the ‘computer on a chip’ that made possible the first handheld calculators and personal computers (PCs). Opportunity/challenge A couple of years back, Intel started to explore the opportunities to leverage the huge amount of data generated in the company over many years. Very soon they realized that with the help of Big Data and predictive analytics, it would be possible to reduce the manufacturing time of their chips. Method Intel has found that there is big value in Big Data. Over the past two years the company has developed more than a dozen data-intensive projects that have bolstered both its operational efficiency and bottom line. Intel runs a huge number of complicated tests on every single chip that comes through the manufacturing process. And while developing new chips, this process helps them uncover lots of bugs and fix them. Every chip Intel makes goes through a quality check, which includes an extensive series of tests. By analyzing historical data collected during manufacturing, Intel can reduce the number of tests it conducts. These tests include some of the information that comes out of the manufacturing process for those pre-release chips, and looking at it; instead of running every single chip through 19,000 tests, Intel is focusing tests on specific chips to cut down test time. Data-intensive processes also help Intel detect failures in its manufacturing line, which is a highly automated environment. Using log files out of manufacturing and test machines across the entire factory network, Intel processes about 5 terabytes an hour. Results By capturing and analyzing this information, Intel can determine when a specific step in one of its manufacturing processes starts to deviate from normal tolerances.This predictive analytics process, implemented on a single line of Intel Core processors in 2012, allowed Intel to save $3 million in manufacturing costs. In 2013–14, Intel extended the process to more chip lines and saved an additional $30 million.4

Big Data case studies  135

Big Data benefits Intel’s security efforts too.The company says its Big Data platform can process 200 billion server events, and can provide early warning of security threats within 30 minutes.

TXU Energy Saving costs by using smart electric meters Background TXU Energy was formally known as Texas Utilities. Since 2002, when the energy market in the state of Texas was deregulated, TXU has been a leading energy supplier in the area. The company has more than 100 years of history. The Dallas Electric Lighting Company, founded in 1882, which started providing energy to Dallas, was an indirect predecessor of TXU Energy. TXU Energy is a market-leading competitive retail electricity provider, powering the lives of more Texans than any other retailer.TXU Energy offers a variety of innovative products and solutions, allowing both its residential and business customers to choose options that best meet their needs, including exceptional customer service, competitively priced  electricity service plans, innovative energy efficiency options, renewable energy programmes and other electricity-related products and services. Opportunity/challenge The energy market is very competitive and with the demand–supply gap, cost is one of the big deciding factors for consumers using or switching their energy supplier. With market deregulation laws winding their way through legislatures around the world, utilities are faced with a new challenge. They must provide excellent service at a low price or customers can switch to an easily available alternative.  Traditionally, energy providers record the energy usage of their consumers monthly and that way they only have the overall usage details of the household customer. However, the rate of electricity usage at different hours of the day varies.This presented an opportunity for TXU to introduce smart meters to their customers to better understand energy usage during the whole day and provide them with informed advice on saving energy and the costs associated with that. Method Smart meters introduced by TXU report near-real-time usage information back to the cloud servers of TXU. Almost every 15 mins, new usage data are

136  Big Data case studies

FIGURE 6.1  Example

demonstration of mobile dashboard

stored in the server and an analytics engine uses this data to map it against cost and utility. Applying detailed analytics to the stream of data from these meters yields tremendous results. For example, analytics can identify individual appliance loads and provide the basis for personalized energy savings recommendations. Consumers can have access to a personalized energy portal and mobile app to see the details, using easy to understand graphs and can follow the recommendations presented to them. To present the right analytics in an understandable format TXU has obtained the help of Bidgely – a startup that enables utility companies to meet their demand-side energy goals by fully engaging and satisfying their customers by presenting data analytics solutions. For this purpose, TXU established a partnership with Bidgely in 2014.5 Figure 6.1 provides a few snap-shots of the mobile app that TXU customers are using to identify the key insights behind their energy usage and save their energy spending.6 With a few swipes, customers can determine: • • • • •

How much energy they are using for the whole house What each major appliance is costing them Their current kWh reading Energy used, or money spent on energy by time of day, and What they can expect to pay for energy at the end of the billing cycle.

Apart from these screens, customers also have access to Home Agent where they can compare their month-on-month bills and electricity usage, so they can see the money they are saving.

Big Data case studies  137

FIGURE 6.2  Example

demonstration of home screen dashboard

Results With the implementation of smart meters and associated analytics dashboard (see Figure 6.2),TXU has given control over energy usage to their customers. Customers feel more informed, are saving money and sticking with the company for longer instead of switching supplier. Using this development TXU has gone ahead and launched a few offerings for customers in order to reshape the demand curve for their customers. One such offering is ‘Free Night Time Energy Charges — All Night. Every Night. All Year Long.’ In fact, TXU promote their service as ‘Do your laundry or run the dishwasher at night and pay nothing for your Energy Charges’. What  TXU Energy is trying to do here is to re-shape energy demand using pricing to manage peak-time demand resulting in savings for both TXU and customers.

138  Big Data case studies

This wouldn’t have been possible without Smart Electric meters and data analytics.

OmedaRx Big Data to improve medication adherence Background OmedaRx is the pharmacy benefit management company of Cambia Health Solutions, parent company of Regence Blue Cross Blue Shield, and manages the prescription plans for Regence health plan members in Oregon, Washington, Utah and Idaho. Many pharmacy companies’ growth depends on the robustness of their health data analytics capabilities that ensure their patients are taking the right medicines and deriving benefits from them. OmedaRx is one such company. However, it is important to dig deep into the data rather than just act on general high-level information. Opportunity/challenges Poor medication adherence is relatively common, according to the U.S. Department of Health and Human Services’ Agency for Healthcare Research and Quality. Studies have shown that 20 to 30 per cent of medication prescriptions are never filled and that, on average, 50 per cent of medications for chronic disease are not taken as prescribed. Poor medication adherence drives $290 billion in avoidable health care costs, or about 13 per cent of total U.S. healthcare expenditures, according to the New England Healthcare Institute, an independent, not-for-profit organization. Traditional methods to determine which patients to reach out to don’t target the people who are at high risk of negative health consequences. With Big Data and more advanced analytics, it’s possible to determine that only 700 are really at risk instead of the intervention of 5,000 people that OmedaRx would have performed. Method The growth and availability of data is key to creating analytics applications that can refine and advance medication adherence programmes. Another key is machine learning, which advances the type of questions that can be asked and answered.  Some time back, OmedaRx began piloting Max for Medication Adherence from analytics provider GNS Healthcare, a company in which Cambia

Big Data case studies  139

Health Solutions is a primary investor. The system uses Big Data to identify individuals at risk of costly drug-related events because they aren’t adhering to the recommended timing and dosage of their medications. The programme starts with OmedaRx sharing the electronic feeds from its pharmacy claims data with GNS Healthcare, which combines it with the electronic feeds from pharmacy and medical claims data from Cambia (and its insurance companies such as Regence BCBS) as well as consumer and demographics information from third-party vendors including information clearing houses. Much of the data OmedaRx sends to GNS Healthcare is structured, but GNS Healthcare can also work with unstructured data. The data is fed into GNS Healthcare’s Max Solution Platform using a number of coding applications capable of handling the unique features of individual data sets. From there, the mixed data is used to develop computational models. The data sets and analytics platform are housed in a GNS Healthcare data warehouse, MeasureBase. Analysts can query the data warehouse using GNS Healthcare’s Measure Language, which uses its own straightforward language that the company says makes it easy to specify what they want to measure, on which people, and over what time periods, in simple terms. GNS Healthcare’s platform can synthesize trillions of data points coming from claims history, electronic medical records, socioeconomic and geographic data, consumer behaviour data, genomics data, bioinformatics data and more. The data is run through GNS Healthcare’s patented Reverse Engineering and Forward Simulation (REFS) machine learning and simulation engine within the Max Solution Platform, which analyzes and models the data sets as multidimensional observations about people over time. The engine learns by reverse-engineering collections of models and then simulates representations to generate predictions, including risks of negative outcomes such as adverse events.7 GNS Healthcare routinely generates analytics reports and models for OmedaRx. The data can be presented in dashboards as well as in stratified lists of individuals and can include risks and other factors resulting from behaviour and clinical behavioural changes. Results OmedaRx’s care management team uses the data to create daily, weekly and monthly outreach programmes that can include phone calls, in-person consultations, emails, etc., to talk with the individuals, better understand their concerns and make recommendations about their prescriptions so they can improve adherence.  Using Big Data analysis, OmedaRx can consider medical and pharmacy claims, history of office visits with physicians, ER visits, hospitalizations, medications prescribed and medication adherence and their own claims data and

140  Big Data case studies

per patient, predict what their risk is of being admitted into the hospital or visiting the ER. Out of those people, OmedaRx can better determine which ones are not taking medications as prescribed.

John Deere Revolutionizing farming using Big Data Background Deere & Company  (brand name  John Deere) is an American  corporation  that manufactures  agricultural,  construction and forestry  machinery,  diesel engines,  drivetrains  (axles, transmissions, gearboxes) used in heavy equipment and lawn care equipment. In 2016, it was listed as 97th in the  Fortune 500  America’s ranking and was ranked 364th in the  Fortune Global 500 ranking in 2016. Since its founding in 1837, John Deere has seen a great many changes in its business, its products, its services. Farming is undergoing a digital revolution. For example, even small-scale farmers are gathering information passively collected by precision agricultural equipment, and many farmers are using information from large datasets and precision analytics to make on-farm decisions. Opportunity Farming has been empirically driven for over a century, but the data collected was not digital. Agriculture Canada’s family of research centres (circa 1920s) meticulously accounted for wheat yields across farms and weather patterns in order to increase efficiency in production. Big Data is different from this historic information gathering in terms of the volume and the analytical potential embedded in contemporary digital technologies. Big Data proponents promise a level of precision, information storage, processing and analyzing that was previously impossible due to technological limitations. Known for its innovative products range, John Deere has realized that if its equipment can start to collect real time data then that data can be used for many purposes in addition to the changes in design and functionalities of the equipment itself. Method John Deere uses Big Data to step into the future of farming to help farmers achieve this ambitious target. In 2012, they released several products that can connect John Deere’s equipment with each other as well as with owners, operators, dealers and agricultural consultants. This interconnectivity helps

Big Data case studies  141

farmers enhance productivity and increase efficiency. John Deere uses sensors added to their equipment to help farmers manage their fleet and to decrease downtime of their tractors as well as to save on fuel.The information is combined with historical and real-time weather data, soil conditions, crop features and many other data sets. The MyJohnDeere.com platform is mobile and iPad enabled, and helps farmers identify farming trends and arrive at decisions on what crop to plant and expected lifecycle and events related to that crop. Using the programming language R, they can forecast seasonal demand, crop yields and estimated revenue for farmers. Results John Deere is delivering huge value to all farmers who use the MyJohnDeere.com platform – there are many benefits, including operational excellence of the machinery in use, better production cycles and access to financial information related to farming. The mobile farm manager gives access to historical as well as real-time field information, performs soil samples from the field and users can share the information directly with trusted advisors for live remote-advice while on the field.

Airbnb Price recommendations using Big Data Background Airbnb was founded in August 2008, is headquartered in San Francisco, California. Airbnb is a peer-to-peer online marketplace and homestay network that enables people to list or rent short-term lodging in residential properties, with the cost of such accommodation set by the property owner.The company receives percentage service fees from both guests and hosts in conjunction with every booking. It has over 2,000,000 listings in 34,000 cities and 191 countries. Pricing in the travel industry is affected by many factors – internal and external and having access to such data can be an asset to travel companies like Airbnb that generate revenue from service providers and customers and it is in Airbnb’s very interest that their room providers should have maximum occupancy. Opportunity It was possible that if Airbnb could use predictive analysis and present realistic recommendations to their hosts regarding price points then it would be very

142  Big Data case studies

FIGURE 6.3  Example

booking screen of Airbnb

FIGURE 6.4  Example

booking screen – calendar view of Airbnb

easy for them to achieve maximum occupancy, thereby Airbnb could generate more revenue without expanding to include more properties and resources. Method Airbnb’s price suggestion engine, which took months to develop and pulls on five billion training data points, has two main components: modelling and machine learning. The model pulls together what Airbnb’s huge data set can

Big Data case studies  143

reveal about a listing’s best price based on things like its neighbourhood and the size of the listing. The Price Tips feature is a constantly updating guide that tells hosts, for each day of the year, how likely it is for them to get a booking at the price they’ve currently chosen. Hosts can glance at a calendar and see what dates are likely to be booked at their current price (green) and which aren’t (red), and they can get price suggestions as well.When hosts price themselves within 5% of the suggested price, they are ‘nearly four times’ as likely to get a booking as when they don’t. The price tips are presented in an easy interface laid over a complex process  – one that crunches  everything from the day of the week to the specific neighbourhood of a listing and surfaces patterns between latitude, longitude and key words like ‘beach’. The value of amenities also varies greatly.Wi-fi makes a listing in San Francisco more likely to be snapped up, whereas hotter climates will see many fewer bookings if they don’t offer air conditioning — though that also depends on the season. Sundays are the least likely day of the week to get a booking, and the rate rises slowly throughout the week until Saturday, the most popular day. Apart from common factors that Airbnb knew to model for. To find even more relationships between listings and the prices they can command, Airbnb developed Aerosolve, a machine learning package that it released in May 2016. Result With Aerosolve,8 Airbnb can surface new patterns that it then uses to better understand what makes a listing command a certain price. For example, the model highlighted that listings at a certain latitude and longitude were commanding good prices and were often using the word ‘sabbia’.Turns out it was Playa del Carmen, a popular beach town in Mexico, and ‘sabbia’ is ‘sand’ in Italian — another piece of information that Airbnb can relay to local hosts in the form of a price tip. With the help of Aerosolve Airbnb has boosted their hosts’ revenue as well as their own topline.

Walmart Smart searching using semantic analysis Background Walmart is an American multinational retail corporation that operates a chain of hypermarkets, discount department stores and grocery stores. Headquartered in Bentonville, Arkansas, the company was founded by Sam Walton in 1962 and incorporated on 31 October 1969.

144  Big Data case studies

Walmart is a family-owned business which was listed on the New York Stock Exchange in 1972. In the late 1980s and early 1990s, the company rose from a regional to a national giant. By 1988, Walmart was the most profitable retailer in the U.S. and by October 1989, it had become the largest in terms of revenue. Geographically limited to the South and lower Midwest of the USA up to the mid-1980s, by the early 1990s the company’s presence spanned from coast to coast – S‌ am’s Club opened in New Jersey in November 1989 and the first California outlet opened in Lancaster in July 1990. Walmart is the world’s largest company by revenue, according to the Fortune Global 500 list in 2016, as well as the largest private employer in the world with 2.2 million employees. Opportunity In 2011–2012 Walmart wanted to make their million online customers have a better shopping experience and increase the top-line revenue from online sales for the company. Method Seeing online retail stores like Amazon and eBay, the Walmart team was very positive that they could provide better customer experience and service online by understanding their customers better and proposing the products and services they may need as per their preferences based on their behaviour during the sales process. A small team at Walmart Lab – the research and technology hub for innovation at Walmart, worked for ten months and developed a search engine that uses semantic search technology to anticipate the intent of a shopper’s search to deliver highly relevant results for them. The technical team at Walmart Lab includes experts in information retrieval, machine learning and text mining with experience from top search and e-commerce companies and renowned research institutions. They named this new search engine Polaris. For any e-commerce site, searching for a shopping result is very different from conducting a general search. Polaris is based on a platform that connects people to places, events and products giving Walmart a richer level of understanding about customers and products.The new search engine uses advanced algorithms including query understanding and synonym mining to glean user intent in delivering results. When a user types in the word ‘denim’, it returns results on jeans, or ‘chlorine tablets’ returns results related to pool equipment. Polaris focuses on engagement understanding, which takes into account how a user behaves with the site to surface the best results for them. It delivers a new and intuitive results page when browsing for topics instead of

Big Data case studies  145

giving a standard list of search results, allowing shoppers to discover new items they may not have considered.  Results In 2012 Walmart.com saw an approximate 10–15 per cent increase in shoppers completing a purchase after searching for a product using the new search engine and during the first quarter of 2013, e-commerce sales increased by over 30%, and the company continued to make strategic investments in the markets that offered the greatest growth opportunity.

Huffington Post Using Big Data to drive the traffic Background The Huffington Post is an online news aggregator and blog, co-founded by Ariana Huffington, a Greek-American author, syndicated columnist and business woman. The site offers news, satire, blogs and original content and covers politics, business, entertainment, environment, technology, popular media, lifestyle, culture, comedy, healthy living, women’s interests and local news. The HuffingtonPost.com, had 22 million monthly visitors to 11 global editions in 2014 and 16 editions planned for the end of 2015. It became one of the most trafficked and fastest growing news and entertainment websites in the world. Opportunity The Huffington Post attracts a lot of people to read the quality articles they write, and this data provides a lot of insights for the company to refine their offerings and increase their revenue. Key areas like quality content, authenticated comments, new revenue sources and giving best value to advertisers were on the top of the company’s list. Method The Huffington Post has used different Big Data analytics techniques to take advantage of this opportunity. HuffPost content garnered 300 million reader comments in 2013. The high volume but sometimes low quality of comments drove them to seek a smart solution for this. They conducted a  conjoint analysis, that is, a statistical technique used to determine how people value different features of a product or service. In the case of HuffPost comments, the conjoint analysis was employed to

146  Big Data case studies

determine the quality of comments coming from an anonymous person or those who have identified themselves either by name or by avatar and from specific geographies. The analysis determined that the quality of the comments was markedly higher from those who were regionally closer to those surveyed, and from those who were not anonymous. They also used Big Data techniques to optimize the content they were producing and refining at the time of content delivery. Huffington Post, like many other publishers, does split or A/B testing on its content and designs. The publisher also uses timing and response rates as data points in analyzing responses to articles and makes appropriate business decisions on that basis. HuffPost uses Big Data to determine how successful new advertising revenue models are. The company developed the Partner Studio, an arm of the advertising department that handles native advertising, or content marketing. HuffPost partners with major brands to create sub-websites that are content oriented, leveraging their writers, editors and designers. Result As part of this project HuffPost can clearly achieve many goals and generate better content, more engaged readers and better revenue for the company.The analysis determined that the quality of the comments was markedly higher from those who were regionally closer to those surveyed, and from those who were not anonymous. The findings drove HuffPost to require commentator registration and disallow anonymous postings. HuffPost partners with major brands to create sub-websites that are content oriented, leveraging their writers, editors and designers. One of the HuffPost’s parent blogs titled ‘10 Ways Living with a Toddler is Like Being in Prison’, received 24,000 views in 7 hours, and 41 per cent of the views were driven by social media referrals.9

Summary This chapter was about learning from others – how other businesses are using Big Data to create value and we have covered a wide variety of brief case studies with two key intentions: 1) B  ig Data is and can create huge value in any vertical or any business unit of a business as long as practising managers are equipped with the right resources. 2) Working with Big data is not difficult – as long as you have fundamental learning, then a business model like C-ADAPT can help you achieve the goals you want from your next Big Data project.

Big Data case studies  147

We have covered a variety of cases in this chapter: Ooredoo has used big data analytics for rebranding and PR purposes, Domino’s Pizza has achieved improved top-line revenue, Tesco is using it for better customer profiling and Delta Airline has solved the most annoying problem of air travel – lost baggage tracking. There are many established and old players using Big Data for value creation – Intel has used Big Data for speeding up the manufacturing process and achieved huge cost savings in this, TXU has introduced smart meters to help their customers save money. Walmart has improved top-line revenue using smart search engines on the basis of Big Data analytics. And then we saw a few newer players who are not only using Big Data analytics for their business but are generating huge amounts of unstructured data themselves – Facebook and Huffington Post. Readers will have a wider exposure after reading these cases and would be able to implement this learning in their future Big Data projects.

Notes  1 https://vimeo.com/channels/jackcasestudies/72907934  2 www.forbes.com/sites/bernardmarr/2016/04/06/big-data-driven-decisionmaking-at-dominos-pizza/2/#4320a7127a17  3 http://news.delta.com/delta-introduces-innovative-baggage-tracking-process-0  4 www.informationweek.com/software/information-management/intel-cutsmanufacturing-costs-with-big-data/d/d-id/1109111  5 w ww.txu.com/about/press-releases/2014/20141106-txu-energy-bidgelypartnership-elevates-customers-ability-to-track-consumption.aspx  6 www.bidgely.com/blog/customer-retention-simple-ideas-that-make-customershappy/  7 w ww.healthdatamanagement.com/news/big-data-helps-omedarx-improvemedication-adherence  8 www.forbes.com/sites/ellenhuet/2015/06/05/how-airbnb-uses-big-data-andmachine-learning-to-guide-hosts-to-the-perfect-price/#d3ae544248d7  9 www.huffingtonpost.com/mike-julianelle/10-ways-living-with-a-toddler-islike-being-in-prison_b_3726549.html?guccounter=1

7 WHAT PRACTITIONERS SAY

I hear and I forget. I see and I remember. I do and I understand. – Confucius

In any field of work, practical experience and its learnings are considered to be more important than theoretical studies and open research. Though all stakeholders add a lot of value in shaping this ‘Big Data economy’, in this chapter we will include the views of Big Data professionals who have been there, done that. So far, in previous chapters, we have understood data, what Big Data is, we have seen Big Data models and worked on cases to implement that, we have also seen case studies from across verticals, business areas and across the globe, and how Big Data can add tremendous value in almost all verticals and business units. Big Data can still be considered as fairly new for many businesses, but there are many market leaders who have already taken the plunge and are leading the way by implementing and improving from their learning. We have already lookd at some cases. In this chapter, we look at the point of view of practitioners about their first encounter with Big Data, the value and challenges they can see on this route and how they envision the Big Data era in future. We reached out to many business managers and leaders via the Big Data survey we conducted with the help of the Ashridge Business School and also conducted personal interviews and curated views from other sources. We received over 200 responses from business managers, working in different geographical locations and with varied organizations from Startups to large Corporates – we had to be selective, so we have presented a holistic view and interviews from complementary backgrounds and organizations. It is interesting to see that everyone is at a different stage of this revolution – some of them have already started to see big value from Big Data but collectively most of them agree that Big Data is the next big thing and are optimistic about its future and how it can add value to their businesses.

What practitioners say  149

We have discussed their opinions about the state of Big Data analytics and the value creation views, processes and projects in their respective organizations. Collectively their views paint a very practical and accurate picture across industries and present a very realistic view to any manager who is planning to take a plunge into the world of Big Data. We are sure that their views will inspire readers in starting their Big Data journey with confidence.

Big Data is important – very important! Without Big Data analytics, companies are blind and deaf, wandering out onto the web like deer on a freeway. – Geoffrey Moore1

By now is it very evident that Big Data is very important for most businesses now – some of them have already started to act on these opportunities and others are preparing to start these projects in the near future. Apart from operational excellence and increasing top line revenue, Big Data applications are also fuelling innovative solutions to otherwise old and boring industries. Founded in 1999, Cornerstone sells customers human-resources software that they can access via its cloud data centres on a subscription basis, and takes in $280 million in annual revenue. The company currently counts Hitachi, Princess Cruise lines, The Ohio State University and Fruit of the Loom among its roughly 2,100 clients.2 Cornerstone OnDemand thinks Big Data can tell you who to hire. Cornerstone OnDemand has unveiled a new Big Data analytics service that will help organizations solve the employee hiring problem. This new Big Data analytics service will help companies analyze the huge amount of Big Data generated about employees to learn something meaningful. ‘We have seen the talent management market advance over the last decade, as global organizations shifted from a focus on process automation through client server solutions to a focus on employee engagement through a consumerized, unified talent management suite. Now we are moving to the next generation of data-driven talent management in a world of Big Data,’ said Adam Miller, president and CEO of Cornerstone OnDemand. Satnav firm TomTom uses Big Data in a big way. The mapping company has integrated Big Data to improve its services for over a decade and doesn’t plan to stop there. ‘We are good at Big Data. We started before everyone else did and this is why we lead the way today; we address real needs with new innovation. We don’t know whose phones, of course, but that doesn’t matter: if any traffic

150  What practitioners say

jam occurs, anywhere, we see hundreds of phones and navigation devices in a row slowdown from a mile per minute to a crawl. Seconds after that, we start helping TomTom customers that are still, say, 20 miles away to detour,’ says TomTom co-founder Pieter Geelen. In the future, TomTom is expecting that the advances in technology around Big Data will make the sharing of this data much easier, and not just between cars and data centres, but between cars and other cars, thus pushing forward the development of self-driving cars. Big Data companies are busy discovering novel Big Data solutions to make the best out of Big Data analytics to grow their businesses. All Big Data applications have become an integral part of our daily lives; however, Big Data is being used in many unconventional ways across different industries. From startup to big established organizations managers are using Big Data for different purposes. Innovation is key, but topline and bottom line revenue is one of the main priorities for established organizations.There is a clear reason for this – they have been generating data for many years, and they also have a set of problems to deal with – in that case it makes sense for them to start using their existing resources in solving current issues. On the other hand, startups often work on the principal of providing something faster, better and cheaper than the existing offerings in the market and because of this they are more likely to use Big Data for innovation and new things. ABI Research3 states that global spending on Big Data by organizations exceeded US$31 billion in 2013, and will reach a staggering US$114 billion in 2018 – and that includes salaries as well as hardware and software – but it is important to note that these numbers don’t include future innovative companies and products that will take shape in coming year – hence it seems that there will be a lot of investment going on in this area. ‘We’re extracting real business value from analytics that take advantage of Big Data. Whether you love Big Data or see it as problematic, clearly it is too big to ignore. But how valuable is all this data and what does it mean to an organization’s success or survival? As a business that uses Big Data for analytics projects internally, and also advises clients about how to use it, we believe it’s a potential gold mine’ (David Williams, CEO, Deloitte Financial Advisory Services LLP). A large part of the managers who participated in the survey that we conducted agree that Big Data is important for their organization. Sixty-eight per cent of managers said that Big Data was an important or very important part of their businesses and twenty-two per cent did not find it important at this stage. Clearly awareness and knowledge of Big Data technologies and how it can be used for creating value among business managers have grown fast in the last few years – this presents the view that Big Data is not just the fad that it was considered to be by some in the past.

What practitioners say  151

Key value from Big Data What do value managers think Big Data can add? It was not surprising to see that many are of the view that Big Data can be used more in improving operational performance, i.e. operational excellence within organizations. Seventy per cent of managers agreed that Big Data analytics can be used for bottom line improvement, hence it could be a great tool for the team to reduce defects, optimize processes, reduce delays and improve response times to customers. Companies like Google and Yahoo! have started to work on Big Data technologies, initially to improve their operations, data storage and information retrieval processes. However, as mentioned earlier, the Big Data revolutions is being led by two types of organizations – established businesses with past and current data, alongside operational issues and new startups and businesses that are using Big Data for innovation etc. Most of the companies we approached in our survey are old established companies and few of them are new and small businesses, hence our data reflects that 70 per cent of managers are looking at Big Data as an opportunity for operational improvement. ‘Big Data analysis brings a number of advantages to the table’, says Dr. Carsten Bange, CEO of BARC. ‘Topping the list of benefits realized from Big Data analysis are better strategic decisions (69%), improved control of operational processes (54%), better understanding of customers (52%) and cost reductions (47%)’. Some businesses have always been built on data. For others, the path to competitive advantage through data and analytics is a new one. In every organization, however, CEO support and involvement are key in setting direction. ‘The stronger the relationship between a CEO and his/her data analytics team, the more likely an organization is to invest in data-centric technologies,’ says Matt Ariker, chief operating officer of the Consumer Marketing Analytics Centre at McKinsey & Company. Around 57 per cent of managers have suggested that Big Data and analytics can provide cost leadership to the organization – clearly optimized operations and efficiency can help organizations lead in reducing costs and building competitive advantage around it. If an organization is a lower-cost producer and won’t compromise on price it will result in higher average gross margins eventually. Big Data gives capabilities to identify, collect data for and measure every aspect of a business so managers can optimize processes and reduce costs. It is not entirely true that established organizations only focus on operational excellence. One of the oldest industries such as healthcare are innovating processes to optimize resource planning and improve bottom-line revenue for the company. Enabling healthcare professionals to capture and analyze mountains of digitized healthcare data to obtain new insights, Big Data cloud platforms are

152  What practitioners say

ushering in a new era of high-quality patient care at lower costs. According to a report by McKinsey & Company, in future Big Data could save Americans $450 billion annually. Part of that future has already arrived – one of the best ways to curb healthcare costs is to keep patients from entering the hospital in the first place. Using new data tools that send automatic alerts when patients are due for immunizations or lab work, more and more physicians are able to reduce hospitalizations by practising better preventive care.  Parkland Hospital in Dallas Texas has been using analytics and predictive modelling to identify high-risk patients in their coronary care unit and predict likely outcomes once patients are sent home. Thus, Parkland has reduced 30-day re-admissions back to Parkland and all area hospitals for Medicare patients with heart failure by 31 per cent. For Parkland, that represents an estimated savings of $500,000 annually, not to mention the savings patients realize by avoiding re-admission. Not only are hospitals saving money by proactively helping their patients to be healthy and avoid coming to hospital, they are also using their unused resources because of this, in serving new customers, thus improving top-line revenue. A new study published in the July 2014 issue of Health Affairs shows how  Big Data analytics is helping pave the way toward reduced costs. ‘The examples we present in this study provide key insights to the “low hanging fruit” in healthcare Big Data and have implications for regulatory oversight, offer suggestions for addressing privacy concerns and underscore the need for support of research on analytics,’ said David Bates, MD, chief quality officer at Brigham and Women’s Hospital and lead author on the study. The data many banks are looking at is structured and semi-structured and include website clicks, transaction records, bankers’ notes and voice recordings from call centres. The study conducted by Thomas Davenport and Jill Dyché of International Institute for Analytics, said that the banks are getting better at understanding common journeys, monitoring for quality of service and identifying reasons for attrition.4 They are monitoring customer ‘journeys’ through the tangle of websites, call centres, tellers and other branch personnel to understand the paths that customers follow through the bank, and how those paths affect attrition or the purchase of particular financial services. Bank of America has also modified its structure to make Big Data more effective, the report said. ‘The bank has historically employed a number of quantitative analysts, but for the Big Data era they have been consolidated and restructured, with matrixed reporting lines to both the central analytics group and to business functions and units. The consumer banking analytics group, for example, made up of the quantitative analysts and data scientists, reports to Aditya Bhasin, who also heads Consumer Marketing and Digital Banking. It is working more closely with business line executives than ever before.’

What practitioners say  153

When it comes to improving the customer experience, a lot of companies do work behind the scenes. Car manufacturer BMW Group has started using Big Data gathered from dealerships and manufacturing outlets all over the world. For example, BMW’s prototype cars produce an average of 15,000 data points from the engine and transmission down to the suspension and brakes. With Big Data analytics, BMW Group can detect and even fix vulnerabilities that show up during the manufacturing process. What normally would have taken months to analyze can now happen in just days. Using data from test drives and other processes, faults can be found and fixed before new cars go into full production. By taking these extra steps, BMW Group improves customer satisfaction by producing higher quality cars. These cars will need to spend less time in the shop, and any repairs that are needed can be taken care of in less time. Needless to say, the less time and money customers spend on their cars, the more satisfied they will be. Just recently, Southwest Airlines announced a decision to implement cloud technology for data storage and processing with a new suite of customer contact and workforce optimization solutions provided by  Aspect. This allows Southwest Airlines to provide even better customer service using more data analysis. Front-line personnel will receive real-time KPI dashboards related to operational and strategic goals. They will use speech analytics to extract deep and meaningful information out of live-recorded interactions between customers and personnel. This will deliver Southwest Airlines more information on what customers are looking for and what their experience with Southwest Airlines is. Different metrics will guide personnel in their objective to deliver high-quality service. In addition, Southwest Airlines is able to give customer service another boost by analyzing real-time data such as social media analytics in order to provide their customers with tailored offers. Thanks to Big Data, Southwest Airlines knows what products to promote to which customers when and via what channel. This personalized experience leads to a higher conversion and increased customer satisfaction. Big Data analytics also help you in leading innovation and new product/ service developments, as was suggested by 76 per cent of managers who participated in our survey. Companies like Odimax are generating sales leads on the basis of semantic analysis of social media conversations and presenting high quality sales leads to their clients. Another innovative example comes from IBM’s semantic project on social media conversation about ‘Diabetics’ (see Figure 7.1) – it was evident from the results that in some parts of the world people are using music as a preventive therapy for diabetics – and further research on this innovation cure is in process now.5 During the Football World Cup 2014 final, Germans used Big Data analytics to predict Argentina’s game, how they play and their game plan. They

154  What practitioners say

FIGURE 7.1  IBM’s

semantic project on social media conversation about diabetics

formulated their strategy according to this with the help of Big Data analytics. In cricket, ESPN uses Big Data to predict for a particular bowler the kind of delivery he is going to bowl: is it an in-swinger or an out-swinger? They have also achieved knowledge of the strong zone and the weak zone of a particular batsman by analyzing his previous batting performances. Animals in the wild are difficult to track and the only way this can be done is by leveraging a rudimental tracking system. HP has announced its partnership with Conservation International (CI) for creating Earth Insights through Big Data analytics. The Big Data analytics system is designed to create an early warning system for animals belonging to an endangered species. By using sensors, cameras in the analytics system collect data from close to a thousand devices and then collate it to find out the population of each species. The collected Big Data is then fed into the HP Vertica platform which analyzes the readings precisely to find out regions or locations of endangered species. This helps the conservation department invest time and money in specific regions to save endangered species. Managers have suggested a few more uses of Big Data in their businesses; here are a few of them: --

Improving sales processes and identifying changes in communication preferences

What practitioners say  155

--------

Collecting and using interconnected data points Optimizing the effectiveness of the reach of marketing campaigns Enabling strategic market entry Creating more revenue streams via data and analytics gathered Realizing the impact of employee training Improving research and development results and processes Effective market, competition and industry research

Challenges in implementing Big Data projects It has been evident for many years that one of the biggest challenges in implementing any new technology or system in a business is the lack of resources, especially human resources that can gather, understand and learn the latest skills and techniques quickly and move ahead with implementation. Out of the managers who participated in the survey, 73 per cent agreed that lack of knowledge and expertise is one of the main challenges in implementing Big Data projects. Although it was not clear if this was business knowledge to build strategies or technical knowledge to implement those strategies, we assume that Big Data is a considerably new field of study and dedicated efforts are needed by managers and other team members in acquiring the relevant knowledge. Businesses can obtain help from other leaders in this space such as IBM, Google or other consulting companies or associate with educational institutes that work in Big Data fields. Budget has also been raised as one of the challenges by 57 per cent of managers. In our view, Big Data investment should be considered as a longterm strategic investment by the leadership team and an appropriate budget should be allocated for Big Data projects that can not only help in front line revenue growth but also help in operational excellence and cost leadership. There are many direct and proxy value creation measures which have been explained in Chapter 3 – where managers can create value and persuade their leaders to allocate appropriate budgets for Big Data investment. A few participants, 35 per cent of managers, mainly from Europe, have expressed security as one of the main challenges in implementing Big Data projects. Security issues are mainly organizational issues and should be considered by business managers on a case-by-case basis. Organizations have clearly started to work on building strategies for dealing with Big Data analytics and projects. Seventeen per cent of managers stated that they already have a formal strategy in place to deal with Big Data projects, and 39 per cent have mentioned that their organizations are working on building one now. Forty per cent of organizations see no relevance in such a formal strategy and we think that this number will reduce drastically in the coming years. Big Data and value creation is a big opportunity

156  What practitioners say

for businesses and by choice or by force, most organizations will follow this trend soon. In fact, 65 per cent of our participating managers agreed that Big Data will transform their industry in the next 5–10 years and it will change the way business operates in most sectors, while 11 per cent of managers felt otherwise, and the balance did not say anything; though we do not have details about their industries, we can assume that some industries change more slowly than others. In organizations that are using Big Data today, users report overwhelming satisfaction with their results.When we asked about the results of the projects, from those businesses who have implemented at least one Big Data project by now – 73 per cent of them reported getting expected or better than expected results. Overall, managers in different industries are excited about Big Data and how it can be used to create more value for businesses. These are still early days, but organizations are picking up these new technologies fast and filling the talent and other resource gaps quickly. Educational institutes, online portals and private trainers are also adding a lot of value in filling the talent gap in the market and very soon there will be enough resources in the market. We also feel that with more innovation and more talent availability, overall implementation costs and budget requirements will go down.

Summary In this chapter, we have included the views of the people who are in our readers’ shoes – some of them have already started to work on Big Data projects and others are planning to do that. Our survey results from over 200 participants have clearly indicated that Big Data is very important for their businesses. In most cases the key values delivered or expected from Big Data projects are operational excellence, smart decision making and innovation – we have covered these in much detail in Chapter 4 as well. In this chapter, we have also included a few interviews with practitioners who are using Big Data for themselves or their clients. Overall, the main view of most of these practitioners is that Big Data is a very important and promising part of any business.

Notes   1  www.geoffreyamoore.com/, Writer of Crossing the Chasm.  2 http://fortune.com/2015/05/12/cornerstone-recruiting-data/  3 www.abiresearch.com/  4 w ww.forbes.com/sites/tomgroenfeldt/2013/06/11/banks-use-big-data-tounderstand-customers-across-channels/#440234852f07   5 ‘Converting Big Data into Big Knowledge’, article by Lauren Walker, Europe Sales Leader, Big Data Analytics, IBM Information Management, 2012.

8 CONCLUSION AND DISCUSSION

Big Data is one of the most important aspects of most businesses in this day and age. Some businesses have started to join this revolution and others are preparing for it. This pattern of new technology adoption is not new to us – whether you talk about internet or ERP, you will see that most things start with a slow adoption rate and are eventually picked up by most businesses. In the case of Big Data, there are many who claim that this is the new fad in the market but slowly those arguments are being taken down. There is a reason why some people, if not most, considered Big Data to be the next hype –they have seen the journey of data and analytics in the past and they were not able to see the new scope this ‘Big’ element is going to add to the overall offerings; if you see new things through old glasses, you can only see the old things. Irrespective of the fact, and of the authors’ opinion, that Big Data and analytics are here to stay and will add a lot of value to businesses, it would not be entirely correct to say that this adoption will be fast or even that this revolution will create sustainable value for all businesses. Big Data is an important element of businesses and we cannot ignore that – in fact knowingly or unknowingly most of us are contributing to this revolution as users of devices such as mobile phones, and applications such as Facebook and Twitter etc. But as we have seen in this book, the value created should be measured in the right way and be present in all activities around the business, including things like Big Data analytics. We believe that the hidden insights and value potential in the unstructured part of the Big Data is much more than in the structured data – the clear reason for this is that for more than 40 odd years we have been using business intelligence formally in many industries (and informally for many more years) and from data to analytics to representation, we have experimented and invested a lot on structural data analysis but in most cases we have either ignored or couldn’t include unstructured data in main line analysis. It was only when this was becoming a really big problem for internet companies like Yahoo! and Google, that we turned towards unstructured data

158  Conclusion and discussion

curation and analysis, as we discussed earlier. So mainly it has been just over a decade since some of us started to think about the importance of unstructured data. So this is a fairly new, unexplored and exciting addition to the whole data analytics regime – hence it is a key part that has the potential to create a lot of value. Fortunately (or unfortunately) unstructured data is mostly coupled with structured data, hence the next level of analysis is biased towards unstructured data (and rightly so). In Big Data, there are opportunities on the hardware/storage and retrieval side as well as the software/analytics side – though both sides are equally important, the specific side and its investments are dependent on individual businesses. Because of operational or compliance requirements, financial institutions may have to invest a lot in the infrastructure and security of the data and maybe their own data centre, before moving ahead to implement analytics – and in both cases they can create financial value. On the other hand, many startups are using infrastructure as a service and avoiding a big initial investment, directly focusing on the analytics side of it. In this book we have discussed a lot about the analytics side that presents unprecedented opportunities for value creation. Regarding value creation in any business, there is clear differentiation between financial and non-financial value – but mostly it is assumed that a business’s role is to create wealth for its shareholders before anything else, hence financial value creation takes priority over non-financial. Businesses should consider the soft side of value creation and should use proxy measures to take care of the interests of all stakeholders, but if the key focus of the business is non-financial, then the sustainability of the business will be questionable in the long run. Hence any project in a business should be able to take care of its own financial costs and contribution on top of any other requirement. From that perspective, effective and timely measures should be taken by managers and leaders in the organization. Many businesses that are classified differently are basically data companies, including Facebook and Twitter – irrespective or their popularity, user bases and concept, the single most important asset of these companies is the data they are generating via their users’ activities, and their data directly and indirectly is generating consistent revenue for these companies. Other companies like GNIP (acquired by Twitter in 2014) and Datasift are using others’ data (like that of Facebook or Twitter) and creating value by making these unstructured data more usable for other companies to build their system on. Such companies are taking away big infrastructure requirements of data management and providing services where a small business or startup can directly access the unstructured data in a clean and consistent way without having to invest hundreds of thousands of dollars in Big Data setups etc. These companies, including Facebook and Twitter, are creating huge value

Conclusion and discussion  159

from not only building analytics on the Big Data but also reselling and using the data elsewhere. This huge Big Data opportunity is of a technical nature, obviously, but business managers should not be scared by the technical side of it – you will always have technology teams to help you with the ‘how’ part of the equation, but what is more important is that managers should come up with answers to the ‘why’ and ‘what’ questions.We have written this book for business managers because of the very reason that today’s managers should be equipped with the knowledge and techniques to come up with strategic and commercial reasons as to why they should start any project in their organizations and how they will measure the success. Big Data is the biggest thing in value-addition opportunities for businesses right now and managers should not rely completely on their technology counterparts in making strategic decisions. And if you are one of them, then you will have understood the importance of Big Data and analytics for your business after reading this book. Data and its impact on value creation can be internal or external and businesses should be receptive and inclusive in their approach to creating more value for all the stakeholders of the business.We have discussed the stages of Big Data analytics and how industry has reached this stage along with multiple cases where value was created by organizations. One of the key takeaways from reading this book, as in any management reading, was that the strategic thinking and planning for a Big Data analytics project to create business value is more important than the implementation itself. While suggesting this, we have also presented many cases in a very non-technical way, so a non-technical business manager can think about Big Data analytics without having to worry about the technologies. They can then also lead the project in more strategic manner. Dependency on the technical team will now be limited to the implementation only and business managers can lead projects in more strategic ways. One of the biggest contributions to this book is the C-ADAPT model for Big Data analytics.This model was developed by the authors after working with many businesses in different verticals for different goals.The C-ADAPT model is presented here with a very useful C-ADAPT worksheet that any manager and his team can use to define and successfully execute the data analytics project. Now is your time to shine – we hope that after reading this book, you will have a clear understanding of our C-ADAPT model and you will be able to implement this in order to create sustainable value for your organization. You can use the C-ADAPT worksheet (downloadable from www. creating​value.org.uk) and start to work with your team in identifying their business goal, their related areas, associated data sets and analysts. You can work step by step as discussed in Chapter 7 – and very easily you will be able to define the start of your next Big Data analytics project – your technology team will now be able to work with you more efficiently.

INDEX

A/B testing 64–65 Airbnb 141–143 Apache Hadoop 12–13 behavioural recommendations 8 Big Data 1, 10, 22 Big Data analysis 17 Big Data analytics 55 Big Data case studies 123–147 C-ADAPT 1 C-ADAPT model 4, 88–121 C-ADAPT model of Big Data value creation 89 C-ADAPT worksheet 4, 105–122 challenges in implementing Big Data projects 155 competition data 27 correlation and regression 62–64 creating value 1, 30; acquiring more customers 43; customer retention 44; customer satisfaction 41; data driven innovation 50; higher prices 40; improved forecasting 49; reduced customer acquisition cost 42; supply chain 45 customer data 24 data analysis 14 data analytics techniques 60–64 data driven decisions 5, 10 data mining 16 data visualization 76–84 data warehouse 15 DBMS, RDBMS 11, 18

Delta Airlines 132 Domino’s Pizza 126 employee data 26 ER Diagram 18 ERP 18 Gate Gourmet 129 Hadoop nodes 12 Huffington Post 145 Intel 133 John Deere 140 key value from Big Data 151 marketing and sales intelligence 74 mean, median, mode 66–67 non-financial values 36, 38 OmedaRx 138 Ooredoo (formerly Qtel) 124 operational excellence 59 predictive analysis 84–85 proforma balance sheet 34 proforma income statement 34 relational databases 15 return on invested capital 35

Index  161

semantics, sentiment analysis 72 social network analysis 68 standard deviation 67–68 statistical analysis 61 structured data 17

three Vs 22 TXU Energy 135 unstructured data 18 value drivers 31, 32

Tesco 130 text analytics 73–75

Walmart 143