Analytics and Tech Mining for Engineering Managers 9781606505113, 1606505114, 9781606505106

Table of contents :
Content: Cover
Contents
List of Figures
List of Tables
List of Scripts, Examples, and Outputs
Preface
Acknowledgments
Chapter 1: Tech Mining Using Open Source Tools
Chapter 2: Python Installation
Chapter 3: Python Basics for Text Mining
Chapter 4: Sources of Science and Technology Information
Chapter 5: Parsing Collected Data
Chapter 6: Parsing Tree-Structured Files
Chapter 7: Extracting and Reporting on Text
Chapter 8: Indexing and Tabulating the Data
Conclusions
References
Index
Adpage
Backcover.

Citation preview

Analytics and Tech Mining for Engineering Managers

Create your own Customized Content Bundle — the more books you buy, the higher your discount!

This book offers practical tools in Python to students of innova-

THE CONTENT

THE TERMS • Perpetual access for a one time fee • No subscriptions or access fees • Unlimited concurrent usage • Downloadable PDFs • Free MARC records

tion, as well as competitive intelligence professionals, to track new developments in science, technology, and innovation. The book will appeal to both—tech-mining and data science audiences. For tech-mining audiences, Python presents an appealing, all-in-one language for managing the tech-mining process. The book is a complement to other introductory books on the Python language, providing recipes with which a practitioner can grow a practice of mining text. For data science audiences, this book gives a succinct overview over the most useful techniques of text mining. The book also provides relevant domain knowledge from engineering management; so, an appropriate context for analysis can be created. This is the first book of a two-book series. This first book ­discusses the mining of text, while the second one describes the analysis of text. This book describes how to extract actionable intelligence from a variety of sources including scientific articles, patents, pdfs, and web pages. There is a variety of tools available within Python for mining text. In particular, we discuss the use of pandas, BeautifulSoup, and pdfminer. Scott W. Cunningham is an associate professor at the Delft University of Technology. He teaches and researches topics including data ­science, network science, and game theory. His research is directed toward helping national governments anticipate the potentially ­unforeseen consequences of new and emerging technologies. Prior to joining Delft University of Technology, he worked for AT&T as a knowledge discovery analyst, helping customers in the manufacturing and commercial sectors make the best use of their data. Jan H. Kwakkel is an assistant professor at Delft University of Technology. His research focusses on model-based support for ­decision making, with a particular focus on the treatment of u ­ ncertainty. Text mining, and more general machine learning ­techniques, are

For further information, a free trial, or to order, contact:  [email protected]

­important in his research. He has applied his research in various domains including transport, water, and energy.

ISBN: 978-1-60650-510-6

Analytics and Tech Mining for Engineering Managers

• Manufacturing Engineering • Mechanical & Chemical Engineering • Materials Science & Engineering • Civil & Environmental Engineering • Advanced Energy Technologies

Scott W. Cunningham • Jan H. Kwakkel

CUNNINGHAM • KWAKKEL

EBOOKS FOR THE ENGINEERING LIBRARY

ENGINEERING MANAGEMENT COLLECTION C.M. Chang, Editor

Analytics and Tech Mining for Engineering Managers

Scott W. Cunningham Jan H. Kwakkel

ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS SCOTT W. CUNNINGHAM AND JAN H. KWAKKEL

MOMENTUM PRESS, LLC, NEW YORK

Analytics and Tech Mining for Engineering Managers Copyright © Momentum Press®, LLC, 2016. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means—­ electronic, mechanical, photocopy, recording, or any other—except for brief quotations, not to exceed 400 words, without the prior permission of the publisher. First published by Momentum Press®, LLC 222 East 46th Street, New York, NY 10017 www.momentumpress.net ISBN-13: 978-1-60650-510-6 (print) ISBN-13: 978-1-60650-511-3 (e-book) Momentum Press Engineering Management Collection Collection ISSN: 2376-4899 (print) Collection ISSN: 2376-4902 (electronic) Cover and interior design by Exeter Premedia Services Private Ltd., Chennai, India 10 9 8 7 6 5 4 3 2 1 Printed in the United States of America

SWC—This book is dedicated to my mother, Joan Cunningham.

Abstract This book offers practical tools in Python to students of innovation as well as competitive intelligence professionals to track new developments in science, technology, and innovation. The book will appeal to both—tech-mining and data science audiences. For tech-mining audiences, Python presents an appealing, all-in-one language for managing the tech-mining process. The book is a complement to other introductory books on the Python language, providing recipes with which a practitioner can grow a practice of mining text. For data science audiences, this book gives a succinct overview of the most useful techniques of text ­mining. The book also provides relevant domain knowledge from engineering management; so, an appropriate context for analysis can be created. This is the first book of a two-book series. This first book discusses the mining of text, while the second one describes the analysis of text. This book describes how to extract actionable intelligence from a variety of sources including scientific articles, patents, pdfs, and web pages. There are a variety of tools available within Python for mining text. In particular, we discuss the use of pandas, BeautifulSoup, and pdfminer.

KEYWORDS data science, natural language processing, patent analysis, Python, s­ cience, technology and innovation, tech mining

Contents List of Figures

xi

List of Tables

xiii

List of Scripts, Examples, and Outputs Preface Acknowledgments 1 Tech Mining Using Open Source Tools

xv

xix xxiii 1

1.1 Why This Book

1

1.2 Who Would Be Interested

3

1.3 The State of Play

4

1.4 What Comes Next

5

2 Python Installation

9

2.1 Scripts, Data, and Examples

9

2.2 Different Versions of Python

11

2.3 Installing Python

12

2.4 Development Environment

13

2.5 Packages

17

3 Python Basics for Text Mining

21

3.1 Input, Strings, and Output

21

3.2 Data Structures

26

3.3 Compound Data Structures

30

4 Sources of Science and Technology Information

33

4.1 Collecting and Downloading the Data

34

4.2 Altmetrics and the Supply and Demand for Knowledge

42

4.3 Examples Used in the Text

44

x  •   Contents

5 Parsing Collected Data

47

5.1 Reading Column-Structured Data

47

5.2 Reading Row-Structured Data

51

5.3 Adapting the Parsers for New Databases

53

5.4 Reading and Parsing from a Directory

54

5.5 Reading and Printing a JSON Dictionary of Dictionaries

56

6 Parsing Tree-Structured Files

61

6.1 Reading an XML File

62

6.2 Web Scraping Using BeautifulSoup

67

6.3 Mining Content from PDF Files

71

7 Extracting and Reporting on Text

75

7.1 Splitting JSONS on an Attribute

77

7.2 Making a Counter

78

7.3 Making Simple Reports from the Data

79

7.4 Making Dictionaries of the Data

82

7.5 Counting Words in Documents

83

8 Indexing and Tabulating the Data

89

8.1 Creating a Partial Index of the Data

90

8.2 Making Dataframes

91

8.3 Creating Cross-Tabs

96

8.4 Reporting on Dataframes

100

Conclusions

103

References

111

Index

115

List of Figures Figure 2.1. Figure 2.2. Figure 2.3. Figure 2.4. Figure 2.5. Figure 2.6. Figure 2.7. Figure 2.8. Figure 2.9. Figure 2.10. Figure 2.11. Figure 2.12. Figure 3.1. Figure 3.2. Figure 3.3. Figure 3.4. Figure 3.5. Figure 3.6. Figure 3.7. Figure 4.1. Figure 4.2. Figure 4.3. Figure 4.4. Figure 4.5. Figure 4.6. Figure 4.7.

Data, notebook, and output setup. Anaconda download. Setup wizard. Choose install location. Extracting Anaconda Python. Completing the setup wizard. Accessing the Anaconda command prompt. The command prompt. Type and run iPython Notebook. iPython Notebook server in the browser. New startup directory. Upgrading a package using Pip. Data and output directories. Notebook start page with new directories. New blank notebook. Renaming the notebook. Typing a line in the notebook. Hello world! Simple debug statements. Searching for nanotechnology. Nanotechnology records. Web of science categories. Saving the records. PubMed search interface. PubMed records. PubMed downloads.

11 12 13 13 14 14 15 15 16 16 17 18 22 22 22 23 23 23 24 36 37 37 38 40 41 41

xii  •   List of Figures

Figure 5.1. Figure 5.2. Figure 6.1. Figure 8.1. Figure 8.2. Figure 8.3. Figure 8.4. Figure 8.5. Figure 8.6. Figure 8.7. Figure 8.8. Figure 8.9. Figure C.1.

Example column-structured format. Example row-structured file. Tree-structured file. Dataframe of article ID by year. Expanded dataframe by year. Dataframe of indexed data. Dataframe with filled missing data. Dataframe with organizations. Cross-tab of content by year. Cross-tab of organization by year. Cross-tab of content by organization. The info method. Process for tech mining study.

48 51 62 92 93 94 94 96 98 99 99 101 108

List of Tables Table 1.1. Table 1.2. Table 1.3. Table 4.1. Table 4.2. Table 4.3. Table 7.1. Table 7.2. Table 7.3.

The Journalist’s questions Types of information product Questions and products Sources of science and technology information Most frequently used fields in scientific records Most frequently used fields in patents Sourcing the journalist’s questions Information products Coverage of information products

5 6 7 34 38 39 76 77 87

List of Scripts, Examples, and Outputs Output 2.1. Output 2.2. Output 2.3. Output 2.4. Output 2.5. Example 3.1. Example 3.2. Example 3.3. Example 3.4. Example 3.5. Example 3.6. Example 3.7. Example 3.8. Output 3.1. Example 3.9. Example 3.10. Example 3.11. Example 3.12. Example 3.13. Output 3.2. Example 3.14. Output 3.3. Example 3.15. Example 3.16. Output 3.4. Example 5.1. Output 5.1. Example 5.2.

Default profile location Notebook directory Changing the notebook directory Using pip Upgrading packages with pip Hello World! String printing Opening a file Writing to a file The enumerate function Lists Dictionaries Sorting a dictionary Sorted dictionary Counters Using a counter Adding two counters Fields in a record Storing fields in a record Stored record Another example of fields in a record A second stored record Dictionary of dictionaries Retrieving an article from a corpus Example output from a dictionary of dictionaries Parsing column-structured data Example record from PubMed database Saving a dictionary of dictionaries to JSON

16 17 17 18 18 22 23 24 25 25 26 27 27 28 29 29 30 30 30 31 31 31 32 32 32 48 50 50

xvi  •   List of Scripts, Examples, and Outputs

Example 5.3. Parsing row-structured data Example 5.4. Adapting the parser for a new databases Example 5.5. Reading from a directory Output 5.2. Example output of reading from a directory Example 5.6. Loading and pretty-printing a JSON file Output 5.3. Sample dictionary of dictionaries Example 5.7. Extracting a sample from the dictionary of dictionaries Output 5.4. Displaying part of a sample record Example 6.1. XML to dictionary Output 6.1. Patent stored in a dictionary Example 6.2. Pretty-printing a dictionary Output 6.2. Sample pretty-printed output Example 6.3. Recursively printing a dictionary and its contents Output 6.3. Top of the patent Output 6.4. Cited literature in the patent Output 6.5. Description of the invention Example 6.4. BeautifulSoup example Output 6.6. Scraped HTML sample Example 6.5. Extracting readable text from HTML Output 6.7. Example readable text from HTML Example 6.6. Example use of PDFMiner Output 6.8. Sample PDF output to text Example 6.7. Get outlines method Example 7.1. Splitting a corpus Output 7.1. Results from splitting Example 7.2. Making a counter Output 7.2. Screen output from a counter Example 7.3. The most common method Output 7.3. The top 10 years Example 7.4. Counting authors Output 7.4. Top 10 authors Example 7.5. Counting nations Output 7.5. Top 10 nations Example 7.6. Extracting a dictionary Example 7.7. Loading a JSON Example 7.8. Fetching a field Output 7.6. Sample counter Example 7.9. Counting a field Output 7.7. The most frequent words Example 8.1. Selective indexing

51 53 55 55 56 57 58 58 63 63 63 64 64 65 66 66 68 68 69 70 71 72 73 77 78 78 79 79 79 80 80 81 81 83 83 84 84 85 86 90

List of Scripts, Examples, and Outputs  •   xvii

Output 8.1. Example 8.2. Example 8.3. Example 8.4. Example 8.5. Example 8.6. Example 8.7. Example 8.8. Example 8.9. Output 8.2. Example 8.10. Example 8.11. Example 8.12. Output 8.3. Example 8.13. Output 8.4. Example 8.14. Example 8.15.

Sample counters by records Making a data frame from an index Expanding the data frame Using the head method Filling missing values Organizations of interest Selective organizational search Creating an organization data frame Sizing the data frame The dimensions of a data frame Creating a content by year cross-tab Creating an organization by year cross-tab Creating a content by organization cross-tab The info method Summing a data frame A summed data frame The describe method Saving a data frame

91 92 93 94 94 95 95 96 97 97 97 98 98 100 101 101 101 102

Preface The authors of this book asked me to share perspectives on tech mining. I co-authored the 2004 book on the topic (Porter and Cunningham 2004). With an eye toward Scott and Jan’s materials, here are some thoughts. These are meant to stimulate your thinking about tech mining and you. Who does tech mining? Experience suggests two contrasting types of people: technology and data folks. Technology folks know the ­subject; they are either experienced professionals or trained professionals or both, working in that industry or research field to expand their intelligence via tech mining. They seek to learn a bit about data manipulation and analytics to accomplish those aims. For instance, imagine a chemist seeking a perspective on scientific opportunities or an electrical engineer analyzing emerging technologies to facilitate open innovation by his or her company. The data science folks are those whose primary skill include some variation of data science and analytics. I, personally, have usually been in this group—needing to learn enough about the subject under study to not be totally unacquainted. Moreover, in collaborating on a major intelligence agency project to identify emerging technologies from f­ ull-text ­analyses, we were taken by the brilliance of the data folks—really impressive capabilities to mine science, technology, and innovation text resources. ­Unfortunately, we were also taken by their disabilities in relating those analyses to real applications. They were unable to practically identify emergence in order to provide usable intelligence. So, challenges arise on both sides. But, a special warning to readers of this book—we suspect you are likely Type B, and we fear that the challenges are tougher for us. Years ago, we would have said the opposite—analysts can analyze anything. Now, we think the other way; that you really need to concentrate on relating your skills to answering real questions in real time. My advice would be to push yourself to perform hands-on analyses on actual tech-mining challenges. Seek out internships

xx  •  Preface

or capstone projects or whatever, to orient your tech mining skills to generate answers to real questions, and to get feedback to check their utility. Having said that, an obvious course of action is to team up Types A and B to collaborate on tech-mining work. This is very attractive, but you must work to communicate well. Don’t invest 90 percent of your energy in that brilliant analysis and 10 percent in telling about it. Think more toward a 50–50 process where you iteratively present preliminary results, and get feedback on the same. Adjust your presentation content and mode to meet your users’ needs, not just your notions of what’s cool. What’s happening in tech mining? The field is advancing. It’s hard for a biased insider like me to gauge this well, but check out the website www.VPInstitute.org. Collect some hundreds of tech-mining-oriented papers and overview their content. You can quickly get a picture of the diverse array of science, technology, and innovation topics addressed in the open-source literature. Less visible—but the major use of tech-mining tools—are the competitive technical intelligence applications by companies and agencies. Tech mining is advancing. In the 2000s, studies largely addressed “who, what, where, and when” questions about an emerging technology. While research profiling is still useful, we now look to go further along following directions. • Assessing R&D in a domain of interest, to inform portfolio management or funding agency program review. • Generating competitive technological intelligence, to track known competitors and to identify potential friends and foes. Tech mining is a key tool to facilitate open innovation by identifying potential sources of complementary capabilities and collaborators. • Technology road mapping by processing text resources (e.g., sets of publication or patent records on a topic under scrutiny) to extract topical content and track its evolution over time periods. • Contributing to future-oriented technology analyses—tech mining provides vital empirical grounding to inform future prospects. Transition from identifying past trends and patterns of engagement to laying out future possibilities is not automatic, and offers a field for productive study. • I’d point to some resources to track what’s happening in tech mining as time progresses. • Note the globalization of tech-mining interest. For instance, this book has been translated in Chinese (Porter and Cunningham

Preface  •   xxi

2012)—not expecting many of you to rush off to read it, but it is an indicator of considerable interest in Asian economies pursuing science, technology, and innovation opportunities. And that reinforces the potential of text processing of languages other than English. • Track the scholarly literature. Tech mining analytics and applications splatter across various scholarly fields. Here I note a few pieces from our colleagues. Bibliometrics journals cover analytical advances—c.f., Ma and Porter (2014) and Zhang et al. (2014). Management of technology-oriented journals cover analytics and applications—c.f., Guo et al. (2014), Newman et al. (2014), and Porter, Cunningham, and Sanz (2015). Should you choose open-source or proprietary tools and software? This book advances an open-source strategy for you to learn skills in Python and other open software, especially to apply to open source data resources. I come from the proprietary side—pursuing use of commercial software (VantagePoint 2016), particularly in analyzing leading science, technology, and innovation resources like Web of Science and Derwent Patents. I’d like to say a bit about the pros and cons of each. Open source advantages favor software, data, and learning. In software, there are lots of open advantages. This includes leveraging other’s contributions and free access. In data, free is good, and this is certainly on the upswing in science, technology, and innovation. And finally, there are learning opportunities that offer inherent value beyond the immediate tasks. For instance, you may be learning Python to do other things with it, as well as gaining transferable programming skills. But don’t write off proprietary resources. If better data are available for a price, they may dominate free options. One can waste a lot in cleaning the crappy data and never catch up with the readily available alternative. Without such suitable cleaning, one could be generating “garbage out from garbage in.” Making your own scripts is expensive. If there is already good software, already available, you should use it. And surely, consider combinations. Don’t rule out open source data, just because you’re using proprietary software—for example, MEDLINE offers uniquely rich coverage of the world’s biomedical research and it’s free to all. Conversely, your open source software may enable you to generate particularly valuable CTI by analyzing a premium information resource, such as Web of Science or Derwent Patents—alone or in conjunction with additional open source resources. Combinations increase your potential resources—for example, VantagePoint (proprietary) works

xxii  •  Preface

well with Pajek (open source) to generate science and patent overlay maps to show disciplinary participation in R&D areas under study (c.f., Kay et al. 2014). Alan Porter Atlanta, Georgia July 30, 2015

Acknowledgments SWC—This work was partially funded by a European Commission grant, grant number 619551. JHK—This work was partially funded by the Dutch National Science foundation, grant number 451-13-018.

CHAPTER 1

Tech Mining Using Open Source Tools This book is for readers with a pervasive interest in science, technology, and innovation and those who invest time in analysis in order to get a deeper sense of the underlying trends in human knowledge. The book ­provides the tools to let you monitor and analyze the raw by-products of scientific activity—whether they are scientific articles, patents, web pages, or social media posts. Although the book requires a basic level of p­ rogramming skills in Python, many detailed examples have been provided that can be used as a basis for further learning. In addition, the examples can be ­further adapted and customized to meet the specific needs of the readers.

1.1  WHY THIS BOOK This book is about using open source software. Broadly speaking, software is an open source if its source code is made available. The license of the software specifies that the copyright holder provides the right to study, change, and distribute the software to anyone and for any purpose. There exists a plethora of open source licenses and a complete discussion of details are beyond the purpose of this book. The emergence of open source software is tied to the rise of the ­Internet. However, some of the essential ingredients of open source software existed before this. For example, in the 1920s and 1930s, various motor companies in the United States freely shared patents based on a cross-­licensing agreement. The Internet itself emerged out of the collaborative process adopted in the context of ARPANET for the development of ­communication protocols.

2  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

Nowadays, open source software spans the space from the operating system (e.g., the Linux kernel) all the way to very specialized applications like GIMP (a Photoshop alternative). Moreover, the idea of open source has spread to other domains as well. For example, in electronics, Arduino is based on open source principles, and a website like Wikipedia also used open source ideals. There are many programming languages available. Why are we using Python in this book? There are several reasons why we have ­chosen Python. First, Python is an open source software. The licenses under which most Python libraries are being distributed are quite liberal. Therefore, they can be distributed freely even in case of commercial applications. It is also free, and can easily be acquired via the Internet. Python is platform independent. It runs under Windows, Mac OSX, and virtually all Linux distributions. Moreover, with a little care, programmers can write code that will run without change on any of these operating systems. Second, Python is a general purpose programming language. This means that Python is designed to be used for developing applications in a wide range of application domains. Other domain-specific languages are much more specialized. For example, Matlab, frequently used in engineering, is designed for matrix operations. Being a general purpose programming language, Python can be used for, among other things, string handling, mathematics, file input and output, connecting to databases, connecting over a network and to websites, and GUI development. Python comes with a comprehensive standard library. The library contains modules for graphical user interface development, for connecting to the Internet as well as various standard relational databases, for handling regular expressions and for software testing. Next to the extensive standard library, there are many libraries under active development for scientific computing applications. This scientific computing community is vibrant and actively developing both—cornerstone libraries for general scientific computing and domain-specific libraries. This implies that Python is increasingly being seen as a viable open source alternative to many established proprietary tools that have typically been used in science and engineering applications. Third, the language design of Python focuses on readability and coherence. Quite often, it is hard to read code, even if you have written it yourself a few weeks ago. In contrast, the language design of Python strives to result in a code that is easy to read. Both collaboration and education benefit from the feature of readability. One of the most obvious ways in which Python enforces readability is through its use of indentation

Tech Mining Using Open Source Tools  •  3

to structure code blocks. In many programming languages, blocks of code are in between curly braces. To understand the structure of the code, one has to detect which curly braces belong together. In contrast, Python structures code blocks through the use of indentation. This structuring might take some getting used to but it produces highly readable code. Fourth, Python is a high level programming language. This means that many details related to the exact working of a computer are abstracted away. For example, in Python, a user typically need not worry about memory management. Because many of these details have been abstracted away, a programmer can focus on getting things done, and done quickly. It is not uncommon that performing a given task in Python requires half the amount of code as compared to the same task in a lower level language.

1.2  WHO WOULD BE INTERESTED You might be a practicing scientist, engineer, or a trainee. You might be an analyst working in a research-intensive industry. You might work in a ­government or nonprofit agency, and you might need to evaluate the impact of current research funding. Or you might be a director or vice president who wants to know what is possible given current state-of-theart ­capabilities for analysis. A number of different professionals need to use and know more about technology intelligence and text mining. In fact, this book has a lot in common with data mining and data science. As a result, we sometimes speak of tech mining, which is the specific application of data mining to studying science, technology, and innovation. But we will, on occasion, also use the wold text mining. Text mining is the application of data mining techniques to qualitative data and, more specifically, text. We expect there to be an increasing and fruitful exchange between applied practitioners in tech mining and those who are mining texts (of social media especially) for other business and organizational purposes. Given the examples in this book, the interested reader will want to build and expand upon them for their own s­tandard routines. This book also does not discuss the design of a complete tech mining study to meet a practical need. If data and standardization is important to you, you might consider acquiring proprietary software for analysis, many of which come with standard subscriptions to large databases of science and technology.

4  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

1.3 THE STATE OF PLAY How much scientific data is out there today? According to some estimates, the world exceeded 1 zettabyte of storage capacity in 2014. Rough estimates suggest that 6 exabytes of this data are texts related to research and development activity. That’s written material equal to the contents of 600 Libraries of Congress. Of course a lot more raw scientific data is being collected, which were earlier refined and analyzed in text only. The Industrial Revolution presaged our modern era of change. The world has been managing rapid technological development for a long time now! Is there anything unique about the character of technological growth being experienced today? Development has historically occurred in fits and starts—there are periods of stability and economic growth, periods of stagnation, and also periods of rapid technological change and disruption. We may be entering another such period of rapid technological change, the likes of which have not been seen since the 1950s. The character of this growth will be qualitatively different. For one, it will likely be heavily dependent on computers, data, and the Internet. Second, it will most likely be science based. And third, this is likely to be a period of open innovation—distributed across many parties rather than being concentrated on large government or industrial bureaus. And finally—although perhaps it need not be said—these technological changes will be surprising and unforeseen for many. Novel societal changes demand new techniques for governance and management. Since this new wave of change is likely to involve (or be about) computing and the Internet, it makes sense to have a set of tools utilizing computers and the Internet And, since this new wave of change is likely to be science based, it makes sense that keeping abreast with change will involve monitoring the by-products of science. This includes scientific articles and also patents. The new age of innovation will also be distributed. This means that individuals and organizations cannot count on having necessary knowledge right at hand; in turn, this places a high demand on coordination between various parties. Participation in science and technology is increasingly becoming a highly distributed process of searching social networks, and a highly asynchronous process of reading and writing large repositories of knowledge. The fascinating work by Hilbert (2014) quantifies the format of world’s data. The digital revolution has been a revolution in text. Given Hilbert’s data, we estimate that there is more than 15 exabytes of stored text today. It plays a more important role than ever before. At least for a while, before digital video fully takes hold, text is growing faster, and is a significant part of our computing and communication infrastructure.

Tech Mining Using Open Source Tools  •  5

Our final point in this section is about the monitoring of change. Changes bring both disruption and opportunities. We believe that skills for monitoring technological change will be critical for organizations as well as technology professionals. In the next section, we’ll discuss what open source software brings to the practice of open innovation.

1.4  WHAT COMES NEXT This closing section of the chapter discusses three things—the structure of the book, the case examples used, and the various code examples. In the structure section of the book, we discuss the organization of the book and how this relates to general processes for data mining and data science. In the example section, we discuss various sources of data used to illustrate the core concepts of the book. In the code example section, we discuss where you can find online the full iPython notebooks, which accompany each of the chapters of the book. With these, you can run the examples at home or office, on your own computers. 1.4.1  STRUCTURE OF THE BOOK This book is the first of the two-volume sets on text mining and text analysis. It covers the mining of text, and takes simple approach to lay out a variety of different possible text analysis questions. The approach is based on “journalist’s questions” (Table 1.1). These are the “who, what, when, where, and why” of the technology intelligence world. These questions can be used to appraise the kinds of technological intelligence you can create using your data. As we progress through the book, we will clarify how to source information that can help answer each of these five questions. Information products come in successive levels of complexity. S ­ imple lists, such as a top 10 author list or a top 10 publishing organization list, give quick insight into the data. Tables provide comprehensive input Table 1.1.  The Journalist’s questions Type of question Who What When Where Why

6  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

s­ uitable for further analysis. For instance you can create a set of articles fully indexed by content. Then it becomes possible to filter and retrieve your content. This often reveals surprising relationships in the data. You can also compile organizational collaborations across the articles. Like article indices, these collaboration tables are often inputs to data analysis or machine-learning routines. The final form of information products, which we’ll consider here, are cross-tabs. Cross-tabs mix two or more of the journalist’s questions to provide more complex insight into a question of research and development management. For instance, a cross-tab that shows which organizations specialize in which content can be used for organizational profiling. A decision-maker may use this as an input into questions of strategic ­alliance. The variety of information products that we will be considering in the book is shown in Table 1.2. Lists include quick top 10 summaries of the data. For instance, a list might be of top 10 most published authors in a given domain. These lists should not be confused with the Python data structure known as a list. We’ll be discussing this data structure in subsequent chapters. A table is a complete record of information, indexed by a unique article id or a unique patent id. Such a table might include the presence or absence of key terms in an article. Another example of a table could include all the collaborating organizations, unique to each article. A cross-tab merges two tables together to produce an informative by-product. For instance, we could combine the content and organization tables to produce an organizational profile indicating which organizations research what topics. Our usage of list, table, and cross-tab is deliberately informal here. The table shows the type of question being asked, as well as the form of information product, resulting in a five by three table of possibility (Table 1.3). Although we haven’t created examples of all the 15 kinds of questions and products represented in this table, there is a representative sample of many of these in the book to follow. We now briefly introduce the book to follow. The next chapters, ­Chapters 2 and 3, provide a quick start to the Python programming language. While there are many fine introductory texts and materials on Table 1.2.  Types of information product Type of information product List Table Cross-tab

Tech Mining Using Open Source Tools  •  7

Table 1.3.  Questions and products List

Table

Cross-tab

Who What When Where Why

Python, we offer a quick start to Python in these two chapters. The chapters provide one standard way of setting up a text mining system, which can get you started if you are new to Python. The chapters also provide details on some of the most important features of the language, to get you started, and to introduce some of the more detailed scripts in the book to follow. There is also a chapter on data understanding, which is Chapter 4 of the book. This chapter covers sources of science, technology, and innovation information. There is a wealth of differently formatted files, but they basically break down into row, column, and tree-structured data. During the data mining process, cleaning and structuring the data is incredibly important. We provide two full chapters on the topic, Chapters 5 and 6, where we guide you through processes of extracting data from a range of text sources. Here, especially the differences between text mining and more general data mining processes become apparent. These chapters introduce the idea that text is structured in three major ways—by rows, by columns, and by trees. The tree format in particular leads us to consider a range of alternative media formats including the pdf format and the web page. The book concludes with Chapters 7 and 8, where we discuss producing informative lists and tables for your text data. These chapters walk into the gradually increasing levels of complexity ranging from simple top 10 lists on your data, to full tables, and then to informative cross-tabs of the data. These outputs are useful for both decision-makers as well as for additional statistical analysis.

CHAPTER 2

Python Installation After reading this chapter, you will be able to setup a running installation of Python. With this installation, you will have a working environment where you can experiment with the scripts in the book, and do your own tech mining. An experienced programmer may skip ahead in the book with no loss of generality. In the chapter to follow, we describe how different versions of language have emerged over time. We then discuss the various operating systems and processors for which Python operates. We describe various distributions of Python, and guide through installing Python 3.4 for a Windows 64 bit machine. We discuss the development environment for Python, and guide you through starting up and running iPython Notebook, a popular scripting environment for Python. We provide some simple examples of writing output to the screen, and reading and writing files. We conclude the chapter by discussing various packages for Python, and introduce you to some of the most useful packages for tech mining. First, however, we discuss how to download and set up the scripts, data, and examples of the book.

2.1  SCRIPTS, DATA, AND EXAMPLES We make the scripts in the book open source and thereby available for your use. To find these examples, go to github (Github 2016), which is a collaborative tool for supporting software development teams. There you can find our code and data under the user @swcunningham, in the “Text-Mining-Repository.”

10  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

Create a directory called “iPython Notebook” in your “C:/” drive. Create the following directory structure under this working directory, and place files as follows. C:/ iPython Notebook/ Chapter 3. Python Basics for Text Mining.ipynb Chapter 5. Parsing Collected Data.ipynb Chapter 6. Parsing Tree Structured Files.ipynb Chapter 7. Extracting and Reporting on Text.ipynb Chapter 8. Indexing and Tabulating the Text.ipynb Data/ stopwords.txt Drug Delivery/ Column Format/ drug_delivery_column1.txt Row Format/ drug_delivery_row1.txt [data files from 1 to 5 here] Nano/ Column Format/ nano2011a.txt [data files from 2011 to 2015  here, in a and b] Row Format/ nano_row_sample.txt Tree Format/ nano_patent.xml Output/ Drug Delivery/ drug_delivery_JSON.txt Nano/ brief_index.txt nano_column_JSON.txt nano_index.txt nano_inst.txt If you are an experienced Python user, you may want to customize our input and output directories to better match your own working style. We’ll use the data and notebooks throughout the txt. The outputs will be produced while working. You’ll need the directories, but you can

Python Installation  •  11

Figure 2.1.  Data, notebook, and output setup.

forego downloading the output. The outputs are there for your reference. Your setup, once finished, will look like Figure 2.1.

2.2  DIFFERENT VERSIONS OF PYTHON Python is a high-level, general-purpose language that emphasizes readability of code. As the level is high, you can accomplish a lot with Python in relatively few number of lines of code. Python emphasizes on a v­ ariety of different programming paradigms, meaning it is easy to shape the code to a variety of different programmers’ needs. Python is increasingly becoming the introductory programming language for many university students worldwide. It is also a premiere language for data science, often sharing that honour with the R language for statistical computing or the Matlab toolbox. Python is now at version 3.4, and this latest version is the one that we will be using for the book. Python is nearly 25 years old; naturally, it has grown and evolved since its first creation. Running with the newest version is important because we want the examples in the book to be as up-to-date as possible, and because we want our readers to avail some of the newest features of the language. Working with the newest version of Python (as of publication) presents some challenges and there can be third party incompatibilities as well. Python is available for a number of operating systems including ­Windows, OSX, and Linux. Python also runs on different processors including 32 bit and 64 bit systems. In the material below, we choose one example—Windows 64 bit systems—and describe how to install a working system for Python coding. You will be able to use the remainder of the book even if you don’t follow our choice of processor and operating

12  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

system. Python is standardized across all operating systems and processors and thus, you will not be missing any features if you make a different choice. This setup describes the process using a laptop with Windows 7 Enterprise installed. Other hardware and operating systems are possible; this combination is selected for demonstration purposes.

2.3 INSTALLING PYTHON You can always freely download Python from the main site (Python ­Software Foundation 2015a). Here you will get an installer for Windows, Mac, or Linux that will quickly download and install the basic distribution of Python. On the other hand, you will need to download and install ­various add-ons separately. For that reason we recommend you get a ­comprehensive Python distribution aimed at scientific computing. Python distribution is a third-party packaging of Python, which includes many of the most popular additions to the language. It’s handy to have these additional packages, which we discuss in greater details below. Our recommended distribution is Anaconda (Continuum Analytics 2015). Other possibilities for a distribution—which we won’t discuss here— include Canopy (Enthought Scientific Computing 2015) and Python(x, y) (Pythonxy 2015). Downloading Anaconda requires navigating to the Continuum ­Analytics page, as shown in Figure 2.2 (Continuum Analytics 2015). After clicking on download Python 3.4, a fairly lengthy download will progress. Clicking on the downloaded installer then begins the installation p­ rocess (Figure 2.3). Click through the presets for the installation, including the suggested download location (Figure 2.4). The installer will unpack

Figure 2.2.  Anaconda download.

Python Installation  •  13

Figure 2.3.  Setup wizard.

Figure 2.4.  Choose install location.

v­ arious components of the download and install them in your directory ­(Figure 2.5). The installation then completes the setup (Figure 2.6).

2.4  DEVELOPMENT ENVIRONMENT Now that Python is installed, the next step is to get a working environment where you can code your Python scripts. More generally, an integrated development environment, or IDE, is a set of tools used for producing scripts or software. In this book, we emphasize the use of iPython Notebook. iPython Notebook is an interactive computational environment

14  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

Figure 2.5.  Extracting Anaconda Python.

Figure 2.6.  Completing the setup wizard.

where you can display and interact with your code, combine your code with text and equations, and easily share the notebooks you create with others (iPython Interactive Computing 2015). An alternative to iPython Notebook is the Eclipse IDE in combination with the PyDev plugin (Eclipse 2015). Eclipse is a fully featured language agnostic IDE. The PyDev plugin allows one to use Eclipse for developing and maintaining Python code for a software production environment. The Anaconda distribution comes with iPython and iPython Notebook (iPython 2016). To access the iPython Notebook, go to your setup ­window. Find the Anaconda command prompt (also available at All P ­ rograms ­listing, as shown in Figure 2.7). After clicking, you access the command

Python Installation  •  15

Figure 2.7.  Accessing the Anaconda command prompt.

Figure 2.8.  The command prompt.

prompt itself (Figure 2.8). Type “ipython notebook” (­ Figure 2.9), and the notebook itself starts up. iPython then starts a server on your computer where you can create new notebooks, or serve existing notebooks. Much like a web page, these are accessible from inside your web browser. On our machine, iPython opens up to the directory in “C:/Users/localadmin/Anaconoda.” This location may vary somewhat according to your specific computer and setup. This location contains a lot of unrelated files and executables. Your running iPython server might look a lot like ours (Figure 2.10). To shut this down, as noted in screen, hit Control+C and exit the tab in your browser.

16  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

Figure 2.9.  Type and run iPython Notebook.

Figure 2.10.  iPython Notebook server in the browser.

We recommend that you create a new directory of “notebooks” and work from there instead. In this manner you can create an orderly working directory to contain your data, create and view your notebooks, and capture the various outputs of the tech mining process. In order to do this type >ipython profile create in the Anaconda command window. This creates a new blank configuration file that you can edit. This enables you to better configure the Notebook to meet your needs. To do this, go to the file iPython_notebook_config.py and open it up for editing in Notepad. You can search for this file in your start menu, or navigate to the appropriate directory in Notebook and open up the file. On our machine, this file is located under (Output 2.1). Output 2.1. Default profile location C:/Users/localadmin/.ipython/profile_default/ Once you’ve opened up this file, find the variable notebook_dir ­(Output  2.2).

Python Installation  •  17

Figure 2.11.  New startup directory.

Output 2.2. Notebook directory # c.NotebookApp.notebook_dir = ‘’ Currently this is commented out and set to a blank value. Uncomment the variable and set it to your preferred starting location. We recommend something like “C:/notebooks.” The resultant line looks like the following (Output 2.3). Output 2.3. Changing the notebook directory c.NotebookApp.notebook_dir = ‘C:/notebooks/’ Now, when you open up the Notebook, you’ll get a clean directory in an easy to find and easy to modify location on your machine (Figure 2.11).

2.5 PACKAGES In this section, we’d like to introduce you to some of the packages you’ll be using for tech mining in Python. Before discussing this in more detail, it is useful to introduce some terminology. A module is a piece of code, which can be used to build larger software programs. A script is a piece of code intended for an end user. A package is a convenient bundling of modules for easy use. The term library is not really used within the Python language, but it may be taken to mean a collection of related packages built and maintained by a third party. Such packages work seamlessly with the base language and greatly expand the range of analysis capabilities

18  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

available in Python. You’ve already gained some first-hand experience with the iPython package in the previous section. The NumPy and SciPy packages expand the numerical analysis and scientific computing capabilities of Python. We’ll be using the packages to work with matrices of data (called “arrays” in NumPy). Also highly useful is the Pandas package. The Pandas package provides DataFrames, a very useful object for creating lists and tables of your text data. Matplotlib is a crucial package for visualizing your data. It provides a number of useful graphics and plotting elements, including familiar tools such as the scatter plot and line graph. NetworkX is a package for analyzing and visualizing social networks. And finally Scikit-learn is a package that implements a wide variety of useful statistical and machine-learning models. All these packages come preinstalled with the Anaconda distribution. We’ll discuss additional packages on a case-by-case basis in the chapters ahead. If you haven’t already installed these packages, or if you find that your packages are out of date, then this is what you must do. For example, suppose you wished to install the seaborn package, which lets you mine the social media feed of Twitter. You can type (Output 2.4). Output 2.4. Using pip >pip install twitter in the Anaconda command prompt. Pip is a python package manager. Pip will automatically download the Twitter package (or most packages of your choice) and then makes these packages accessible for use in your own modules. Pip is also already available in your Anaconda distribution. Figure 2.12 shows an example of using Pip to check whether the Twitter package is up to date. Change “Twitter” to any appropriate package name. If you want to upgrade a package such as Twitter to the latest available version (or just to check whether you have the latest version), type (Output 2.5): Output 2.5. Upgrading packages with pip > pip install –upgrade twitter

Figure 2.12.  Upgrading a package using Pip.

Python Installation  •  19

All of this material, in one form or another, is available online for free in the Python Application Programming Interface. A complete tutorial of the Python language, including input and output statements, is also available (Python Software Foundation 2015b). There are some excellent online help groups as well for solving problems related to Python (Stackoverflow 2015). And there are many excellent introductory books to Python. Rather than repeating this material therefore, our goal is to provide you a helpful reference guide that emphasizes the parts of Python you are most likely to need for text mining. In this chapter, we set up a working environment in Python and ­presented some of the essentials for getting started with programming in Python. In the next chapter, we begin with actual scripts in Python. The next chapter, and the chapters to follow, describe how you can parse text, structure the input using data structures, and then save the results for ­further reference.

CHAPTER 3

Python Basics for Text Mining If you followed the previous sections, you now have the iPython Notebook installed and you are ready to begin some simple Python scripts. Or alternatively, you have now set up a development environment of your own choice. If you are already an experienced Python programmer and want to get started with some simple scripts and recipes for tech mining, you should skip to Chapter 3. After completing this chapter, you will know the basics of file input, file output, and writing on the screen. The chapter describes text data—or strings—and how strings are encoded to accommodate a variety of different characters and languages. The chapter introduces some of the most versatile structures for storing and retrieving data in the Python languages and data structures, which you will most certainly need while mining text and technology. Finally, the chapter discusses the essential control statements used in controlling the logic of your scripts.

3.1 INPUT, STRINGS, AND OUTPUT Here’s how we organize our text mining inputs and outputs before scripting them in Python. Open up your home directory for iPython Notebook (as discussed previously). We’ve set this to “C:/iPython Notebook/.” ­Create two new directories—a data directory and an output directory ­(Figure 3.1). You’ll be processing a lot of text, and you’ll want to preserve this text by saving it in a separate directory. This helps to ensure that you don’t accidently corrupt your original sources, and therefore that you can reproduce your analyses whenever needed.

22  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

Figure 3.1.  Data and output directories.

Figure 3.2.  Notebook start page with new directories.

Figure 3.3.  New blank notebook.

Now open iPython Notebook from the command prompt (Figure 3.1). Here you can see the two new directories you created. Create a new notebook by hitting “New” in the upper right corner of the server page. Your newly created notebook will look like Figure 3.2. Rename this notebook by placing and clicking the cursor on the title of the notebook—“Untitled.” Rename the notebook to “Chapter 3—Python Preliminaries” (Figures 3.3 and 3.4). Now let’s run a simple script that outputs “Hello World!” to the screen. Type in the command shown below (Figure 3.5, Example 3.1). Example 3.1. Hello World! print(“Hello World”)

Python Basics for Text Mining  •  23

Figure 3.4.  Renaming the notebook.

Figure 3.5.  Typing a line in the notebook.

Figure 3.6.  Hello world!

Now hit the run button, which is the forward triangle just below the Cell menu. The results are shown in Figure 3.6. Now let’s try that again but output some string variables instead. Type the following in the code block and hit Run (Example 3.2). Example 3.2. String printing greeting = “Hello World!” print(“my greeting is”, greeting) print(type(greeting)) Here we are creating a variable, named “greeting,” and assigning it the value of “Hello World!” Now when we print the variable, the contents

24  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

Figure 3.7.  Simple debug statements.

of the string are output to the screen. We further ask Python to print the “type” of the variable, which is an “str” or “string” variable (Figure 3.7). Note how we can print multiple things as long as they are separated by the comma. Typing out text and variables like this is a handy, if low tech, approach to debugging your scripts. There are a variety of ways in which you can interact with files on your computer. This is relevant both for accessing data and for storing results from your analysis. For most common formats, Python ships with a dedicated library for handling this file. For example, for accessing comma-­separated files, Python comes with a CSV library; for accessing Java Script Object Notation (JSON) files, Python comes with the JSON library. If you are dealing with a file with a well-specified structure, or want to save your data to such a structure, it is recommended to use the dedicated library. However, in some cases, you still need to handle the parsing of the file on your own. The most basic file handling in Python is then your only real option. The most basic way of accessing a file is through the open() function. This function creates a Python file object linked to a file on your machine. You can specify the mode, either read or write, by adding “r” or “w” as an argument to the open function. open() returns a file handle object. This object has several useful methods. Most important are read(), readline(), readlines(), write(), and writelines(). The call to read() reads the entire file into memory; the other method names are self-explanatory. When reading files, it is recommended not to read the entire file into memory in one go. Often, you want to process the content of the file while reading it. A standard way of doing this is as follows (Example 3.3). Example 3.3. Opening a file with open(‘Data/stopwords.txt’, ‘r’) as f: for line in f: print(line.rstrip()) f.close()

Python Basics for Text Mining  •  25

If you are following along with our scripts and data, you may want to download this notebook and the stopwords.txt data file from the accompanying repository on github. This snippet contains a few elements. The first thing to notice is the importance of indentation in the Python programming language, for readability and syntax. The indents signify meaningful blocks of code. We’ll discuss control statements in more detail. Other items to notice is the with statement. This statement creates a context manager, which will make sure the file is closed once you are done with it. We next open the file and associate the file object with the variable name f. Next we iterate over the lines in the file; this is done through the line for line in f. Iterating over the file in this way means that we retrieve one line at a time. The iterator will keep track of where we are in the file; so, we do not have to keep track of the line number. Next we can write dedicated code for handling each individual line. This specific example outputs a list of small words to screen, removing the new lines from the end of each line. An example of writing to a file is shown below (Example 3.4). Example 3.4. Writing to a file with open(‘Output/sample.txt’, ‘w’) as g:   g.write(“Some Text!\n”)   g.write(“Some More Text!\n”) g.close() In a manner similar to the previous example, we open a file, assign it a handle, and then use the write() command to write text. This block executes without any output to the screen, but you will find a new file in the output directory. Another nice thing you can do is to keep track of the items in the data structure you are processing. You do this with the built-in enumerate function (Example 3.5). Example 3.5. The enumerate function with open(“Data/Drug Delivery/Column Format/drug_ delivery_column1.txt”, ‘r’) as f:   for n,line in enumerate(f):    if (n 0):    tag = line[0:4]    separator = line[5]    content = line[6:]    if (tag == “ “):    tag = lasttag    lasttag = tag    tag = tag.strip()    if (tag == “PMID”):    article_id = content    if (tag in record):     contentlist = record[tag]    contentlist.append(content)   else:     record[tag] = [content]   else:     corpus[article_id] = record filehandle.close(), As discussed in the previous example, this is a script that takes a file handle and returns a dictionary of dictionaries. The corpus variable contains a dictionary of the dictionary structure containing all the records parsed. The file is organized so that the first four characters are a field tag. The separator is always a dash and in the fifth position. The actual contents of the field are all characters after the sixth position in each line. The end of the record is signified by a blank line. The row-structured file is characterized by a flow-through across lines. Whenever we encounter a blank tag, we know that we are on a flowthrough line. So we update the tag to the last observed tag and continue processing. If we find the field called PMID, then we record this as the unique identifier for the article. We’ll use that later as a key to store the record in the corpus.

Parsing Collected Data  •  53

The flow-through content requires additional special treatment. For each row in the flow-through, add the content to a list. This enables further specialized processing according to the specific field. We might, for instance, want to handle a list of authors differently than a list of lines in the abstract. Implementing this flow-through list requires the following steps. If this is the first time that the field has been stored in the record dictionary, initialize the dictionary with a field and the contents of the record. The contents of the record become the first elements in a list. If we encounter the record again, we append the new contents onto the end of the list. Once all the lines are processed. we can then store the JSON in a text file for later access and retrieval.

5.3 ADAPTING THE PARSERS FOR NEW DATABASES The last two sections describe parsing PubMed publications. Running the scripts for new databases requires a few modifications. For instance, if we wished to run the scripts for the Web of Science database, you would need to know the unique field identifier, which is contained under UT. You’d need to know how to split the record. Web of Science column-formatted records are split by comma only. You also need to know the end of the record, which is contained in its own empty field under the ER tag. Here is another example (Example 5.4). Example 5.4. Adapting the parser for a new database filename = ‘Data/Nano/Row Format/nano_row_sample. txt’ filehandle = open(filename,‘r’) record = {} corpus = {} for number, line in enumerate(filehandle):   tag = line[0:2]   separator = line[3:3]   content = line[3:]   content = content.strip()   if (tag == “ “):    tag = lasttag

54  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

  lasttag = tag   if (tag == “UT”):    article_id = content   if (tag in record):    contentlist = record[tag]   contentlist.append(content)  else:    record[tag] = [content]   if (tag == “ER”):    corpus[article_id]=record    record = {} filehandle.close(), In the next example, we take a look at growing an existing corpus of text. For various reasons, we may want to integrate a variety of texts into our corpus. Perhaps the download process requires that we store the downloads in multiple files. Or perhaps the query involves fusing different queries into a single file. Or we may be in a process of continuous downloading and updating our file as time goes on.

5.4  READING AND PARSING FROM A DIRECTORY Some databases limit your download capabilities to 500 records at a time. This can create a lot of small files that you need to parse at once. It is better to parse all similar records at once, creating a single corpus that you can use for further reporting or analysis. Another very common use case is to your original database query. Perhaps as a result of a complete text mining cycle, you gain a greater insight into the topic you are analyzing. You may want to go back to the original data and add in new records from nearby related fields. Another nice advantage of incorporating multiple files into a single archive is the ability to eliminate duplicate entries. These duplicate records can easily happen because of user error, or as new records are added to the database while you continue the search over multiple days. The following example describes a script you can use for these purposes. The script begins with a path. As an output, it creates a list of all files it finds relative to this directory path. You can then iterate through the list, parsing each of the files using one of the parsers previously described.

Parsing Collected Data  •  55

When using this function, you should prepare your own directory structure carefully. Be sure to only put the files you want to be parsed in the directory, and no other files. You also should not include directories within your directories, unless you wish to further elaborate the structure described in the following example (Example 5.5). Example 5.5. Reading from a directory importjson fromos import walk allfiles = [] allpaths = [] my_directory = ‘Data/Drug Delivery/Row Format’ for (dirpath, dirnames, filenames) in walk(my_directory):   allfiles.extend(filenames)  print() for file in allfiles:   newpath = my_directory+”/”+file  allpaths.append(newpath) print(allfiles) print(allpaths)

return allpaths

The function initializes a list of files stored in the “allfiles” list. These are the individual filenames found in the root of the directory. The function also initializes a list of paths. These are the complete paths to a single file in the directory. The function calls the walk method. This returns directory paths andnames, and filenames. We only save the filenames, extending them to the end of the allfiles list. Now, for each file in the allfiles list, we create a full relative directory path. These are appended to the end of the allpaths variable. Sample outputs from this script are as follows (Output 5.2). Output 5.2. Example output of reading from a directory [‘drug_delivery_row1.txt’, ‘drug_delivery_row2. txt’, ‘drug_delivery_row3.txt’, ‘drug_delivery_ row4.txt’, ‘drug_delivery_row5.txt’]

56  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

[‘Data/Drug Delivery/Row Format/drug_delivery_row1.txt’, ‘Data/Drug Delivery/Row Format/ drug_delivery_row2.txt’, ‘Data/Drug Delivery/Row Format/drug_delivery_row3.txt’, ‘Data/Drug Delivery/Row Format/drug_delivery_row4.txt’, ‘Data/ Drug Delivery/Row Format/drug_delivery_row5.txt’]

5.5 READING AND PRINTING A JSON DICTIONARY OF DICTIONARIES As we close out the chapter, we discuss saving and loading files in the JSON format. It provides a means for serializing, or storing, complex data structures such as those discussed in this chapter. We also give an e­ xample of iterating through a JSON file, showing the file contents. We’ll use this information later while assembling lists, reports, and other descriptive information about our corpus. In this short example, we import the JSON and pprint libraries. We point to the file we want to open. Then the load command opens the text data into a dictionary of dictionaries. This is pretty printed and finally, the file is explicitly closed (Example 5.6). Example 5.6. Loading and pretty-printing a JSON file from pprint import pprint inputfile = ‘Output/Drug Delivery/drug_delivery_ JSON.txt’ with open(inputfile) as json_data:  dd=json.load(json_data)  pprint(dd) json_data.close() Sample output from the script is shown in the following block of text. The dictionary data structure is wrapped in curly brackets, { }. The open bracket is shown, but as we have cut the output short, the ends of the brackets are not shown. Then, each article is given in the form of a key-value pair. The key is the unique article identifier, or in this case, “PMID:26717200.” In subsequent runs, this unique identifier might be different, as the dictionary is not stored in any particular

Parsing Collected Data  •  57

record order. After the colon, the main body of information about the article begins. There is only one partial entry in this corpus because we have truncated the output to the screen. Because the article contents are stored in a dictionary, the main body of the record is wrapped in an inner set of curly brackets, {}. The content of the article contains further lists, for instance, lists of text or of author names. As an example, the first listed key in the article dictionary is the “Db” tag, which contains the “pubmed” descriptor. The second item “Description” contains all the authors of the article. The “Details” field contains the name of the journal, the date, and some information about the electronic publication (Output 5.3). Output 5.3. Sample dictionary of dictionaries {‘PMID:26717200’: {‘Db’: ‘pubmed’, ‘Description’: ‘Li F, Wu G, Zheng H, Wang L, Zhao Z.’, ‘Details’: ‘Eur J Med Chem. 2015 Dec 15;108:486-494. doi: ’   ‘10.1016/j.ejmech.2015.12.021. [Epub ahead of ’   ‘print]’, ‘EntrezUID’: ‘26717200’, ‘Identifiers’: ‘PMID:26717200’, ‘Properties’: ‘create date:2015/12/31 | first author:Li F’, ‘Resource’: ‘PubMed’, ‘ShortDetails’: ‘Eur J Med Chem. 2015’, ‘Title’: ‘Synthesis, colon-targeted studies and ’   ‘pharmacological evaluation of an anti-­ ulcerative ’   ‘colitis drug 4-Aminosalicylic ’  ‘acid-β-O-glucoside.’, ‘Type’: ‘citation’, ‘URL’: ‘/pubmed/26717200’}, In the following example, we extract three principal content fields out of a sample record. We use a single record just as an example. Let’s specify our search to be article id “PNMID:26717508.” The sample content fields are the title, the journal, and the authors. The title is contained

58  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

in “Title,” the journal is contained in “ShortDetails,” and the authors are contained in “Details.” An example follows (Example 5.7). Example 5.7. Extracting a sample from the dictionary of dictionaries with open(‘Output/Drug Delivery/drug_delivery_ JSON.txt’) as data_file:   data = json.load(data_file) key_set = data.keys() sample_id = “PMID:26717508” article = data[sample_id] sample_title = article[‘Title’] sample_journal = article[‘ShortDetails’] sample_author = article[‘Description’] print(“The sampled title is:”, sample_title) print() print(“The sampled journal is:”, sample_journal) print() print(“The sampled authors are:”, sample_author) print() The results from the script are shown as follows (Output 5.4). Output 5.4. Displaying part of a sample record The sampled title is Rheological behavior and Ibuprofen delivery applications of pH responsive composite alginate hydrogels. The sampled journal is Colloids Surf B Biointerfaces. 2015 The sampled authors are Jabeen S, Maswal M, Chat OA, Rather GM, Dar AA. This chapter discussed several important work-flow issues in the text-mining process. We discussed how to extend an existing corpus of data with new records and simultaneously parse a number of files in a directory, all at once. A final and important lesson of the chapter is the necessity of closer examination and introspection of the raw data and the stored JSON formats.

Parsing Collected Data  •  59

In the following chapter, we will discuss a third format for parsing files that are in a tree-structured format. Many documents are also stored in a tree format, so it is natural to take a look at mining pdf documents, as well as web pages in this chapter. We also provide a patent example stored in the XML format. This discussion concerning XML addresses yet a third format, complementing the discussion of the row and column formats as discussed in this chapter.

CHAPTER 6

Parsing Tree-Structured Files There are two high-level objectives of this chapter. The first is to discuss character encoding. Until now, we have discussed text as if there is a single and universal way to represent text on our computers. In a world that is multinational and multilingual, it is important to understand the nature of text encoding and, second, to be able to handle various encodings when doing text mining. The second goal is to expand our repertoire of parsing routines. We’ve given widely applicable examples of how to parse row-structured and column-structured files. Now it is time to turn to tree-structured data formats. Many richly annotated forms of media are stored in a tree-structured format. Many of these media are highly relevant for monitoring science and technology. Web pages, such as news sites and wiki pages, provide valuable information. Home pages of firms are also of strong interest. In addition, a lot of science and technology contents are in the form of pdf files. Until recently, the pdf format has not been very accessible for machine reading. This chapter provides you tools useful for mining the pdf files. We’ll be discussing the BeautifulSoup, pdfminer, and xmltodict packages in the examples to follow. These are all packages specifically adapted to the needs of reading tree-structured files and formats. Let us first ­discuss tree-structured files. Trees have nodes and links. The terminology of trees resembles that of a family, so that there are parent and children nodes. Two children of the same parents are siblings. Children can also be parents, with their own children, extending the tree to multiple layers. Links between elements are preserved by container elements such as lists or dictionaries. If the children are to be accessed for any special purpose, or if the data is well-structured,

62  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

a dictionary is often used. Otherwise a list is more commonly seen. More rarely, you may find children for whom the ordering must be preserved. More specialized structures including tables are then seen. In another departure, in this chapter we address parsing the whole text instead of just an abstract. Whole texts are usually stored and structured as a tree. Much like an outline, a text tree contains sections and subsections, each embedded in the whole text. In addition whole texts often contain metadata or other nonreadable elements that are used in rendering the ­document. Whole texts also warrant new styles of analyses.

6.1  READING AN XML FILE This section describes reading and parsing an Extensible Mark-up Language (XML) file. XML is a way of annotating a document so that it can be used for both human and machine use. As noted earlier, this is a tree-like record format, which is often used in patent databases. Our e­ xample here comes from a publically available patent database, the European P ­ atent Office (EPO 2016a). This example parses a single nanotechnology application patent, issued by the European Patent Office (Figure 6.1). The patent is stored in an XML format, which we will parse and systematically explore. There are many packages available for parsing XML files. The packages come with varying degrees of power and ease of use. For our purposes we will use the xmltodict package. This package simply and easily converts XML format to a nested dictionary (Example 6.1).

Figure 6.1.  Tree-structured file.

Parsing Tree-Structured Files  •  63

Example 6.1. XML to dictionary importxmltodict with open(“Data/Nano/Tree Format/nano_patent. xml”) as fd: d = xmltodict.parse(fd.read()) The XML file has been restructured to a compound data structure. Here is a small example of what the new data structure contains (Output 6.1). Output 6.1. Patent stored in a dictionary OrderedDict([(‘ep-patent-document’, OrderedDict([(‘SDOBI’, OrderedDict([(‘@lang’, ‘en’), (‘B000’, OrderedDict([(‘eptags’, OrderedDict([(‘B001EP’, ‘ATBECHDEDKESFRGBGRITLILUNLSEMCPTIESI....FIRO..CY..TRBGCZEEHU..SK...... ..............................’), (‘B003EP’, ‘*’), (‘B005EP’, ‘J’), (‘B007EP’, ‘DIM360 Ver 2.15 (14 Jul 2008) -2100000/0’)]))])), (‘B100’, OrderedDict([(‘B110’, ‘1483737’), (‘B120’, OrderedDict([(‘B121’, ‘EUROPEAN PATENT SPECIFICATION’)])), (‘B130’, ‘B1’), (‘B140’, OrderedDict([(‘date’, ‘20110309’)])), (‘B190’, ‘EP’)])), (‘B200’, OrderedDict([(‘B210’, ‘03711511.0’), (‘B220’, OrderedDict([(‘date’, ‘20030311’)])), (‘B240’, OrderedDict([(‘B241’, OrderedDict([(‘date’, ‘20040917’)])), (‘B242’, The patent is richly annotated with a lot of data. This is a virtue for finding specific pieces of information that are cleanly documented. For the purposes of this example, we just want to print out the patent clearly so that we can see more of what it contains. The pretty print capability of the base Python language accomplishes some of this. In fact, let’s take a closer look at this pretty printed data structure. Here is an example of printing the patent dictionary, with sample ­output. Here is the command for pretty-printing (Example 6.2). Example 6.2. Pretty-printing a dictionary importpprint as pp pp.pprint(d)

64  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

And here is the sample output (Output 6.2). Output 6.2. Sample pretty-printed output OrderedDict([(‘ep-patent-document’,  OrderedDict([(‘SDOBI’,   OrderedDict([(‘@lang’, ‘en’),    (‘B000’,     OrderedDict([(‘eptags’,      OrderedDict([(‘B001EP’, ‘ATBECHDEDKESFRGBGRITLILUNLSEMCPTIESI....FIRO.. CY..TRBGCZEEHU..SK.............................. ......’), (‘B003EP’, ‘*’), There is of course a lot more to this patent that we have not shown here. At least the pretty print makes good use of indenting to display ­contents. But the data structure types and brackets are unnecessary for understanding the content. Here is a better answer for printing out the results, using a recursive function (Example 6.3). Example 6.3. Recursively printing a dictionary and its contents import re defprint_dict(dictionary, ident = ‘’, braces=1):   “”” Recursively prints nested dictionaries.”””   for key, value in dictionary.items():    if is instance(value, dict):    print(‘%s%s%s%s’ %(ident,braces*’’,key, braces*’’))     print_dict(value, ident+’ ‘, braces+1)    elifisinstance(value,list):     for el in value:     if is instance(el,dict):       print_dict(el, ident+’ ‘, braces+1)     else:      print(‘%s%s%s%s’ %(ident,braces*’’,el, braces*’’))   else:    if is instance(value,str):

Parsing Tree-Structured Files  •  65

    value = re.sub(‘\n’,’’,value)     print(ident+’%s = %s’ %(key, value)) The function takes a dictionary object as input. It then iterates through each of the keys and values in the dictionary. For each of the values, it checks to see what type of object the value is. There are different clauses for handling a dictionary, a list, and a string. In many of these cases, the function calls itself again to do further processing. This enables the function to manage nested objects such as a dictionary of dictionaries of dictionaries. The function adds indents to its print out, appropriate to the depth it has traversed in the tree. Note the structured use of the print statement where first it lays out the print statement, and then it fills in the variables to be printed out. A final note involves the use of a regular expression in this statement. The regular expression strips out the carriage returns that have crept into the address information in this patent. We will be discussing regular expressions and text processing in greater detail in a companion book and chapter. The resultant patent, printed out recursively, looks like the following (Output 6.3). Output 6.3. Top of the patent ep-patent-document SDOBI @lang = en B000 eptags B001EP = ATBECHDEDKESFRGBGRITLILUNLSEMCPTIESI....FIRO..CY..TRBGCZEEHU..SK............... ..................... B003EP = * B005EP = J     B007EP = DIM360 Ver 2.15 (14 Jul 2008) - 2100000/0 B100 B110 = 1483737 B120 B121 = EUROPEAN PATENT SPECIFICATION On skipping down further, the patent describes the cited literature (Output 6.4).

66  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

Output 6.4. Cited literature in the patent  b = Non-patent literature cited in the description @id = ref-p0001 @num = i = This list of references cited by the applicant is for the reader’s convenience only. It does not form part of the European patent document. Even though great care has been taken in compiling the references, errors or omissions cannot be excluded and the EPO disclaims all liability in this regard. @id = ref-p0002 @num = ul @id = ref-ul0001 @list-style = bullet patcit @id = ref-pcit0001 @dnum = US6128214A On paging down further, it also provides a detailed description of the invention. Some of this text is given as follows (Output 6.5). Output 6.5. Description of the invention #text = DETAILED DESCRIPTION OF THE INVENTION @id = p0001 @num = 0001 #text = The present invention generally relates to nanotechnology. The present invention also relates to neural networks and neural computing systems and methods thereof. The present invention also relates to physical neural networks, which may be constructed based on nanotechnology. The present invention also relates to nanoconductors such as nanotubes and nanowires. @id = p0002 @num = 0002 #text = Neural networks are computational systems that permit computers to essentially ­ function

Parsing Tree-Structured Files  •  67

in a manner analogous to that of the human brain. Neural networks do not utilize the traditional digital model of manipulating 0’s and 1’s. Instead, neural networks create connections between processing elements, which are equivalent to neurons of a human brain. Neural networks are thus based on various electronic circuits that are modeled on human nerve cells (i.e., neurons). Generally, a neural network is an information-processing network, which is inspired by the manner in which a human brain performs a particular task or function of interest. Computational or artificial neural networks are thus inspired by biological neural systems. The elementary building block of biological neural systems is of course the neuron, the modifiable connections between the neurons, and the topology of the network. Processing the patent further could go in several directions. We could strip out specific key value pairs corresponding to the inventors or assignees, or we could seek out the classification codes from the patents. Another interesting avenue would be to assemble networks based on the provided patent and nonpatent literature. Alternatively, we could mine the description of the invention to learn more about the semantics behind the technology, and to draw comparisons across patents. The analysis could easily be replicated across thousands of download patents in a corpus.

6.2  WEB SCRAPING USING BEAUTIFULSOUP Web scraping is greatly facilitated by the use of BeautifulSoup library. Web pages can be static in character, or generated on the fly according to the needs of the user. Either way, the content of the page is transmitted across the page using an HTML document. Every well-formed HTML document can be stored or analyzed in a tree-like data structure. BeautifulSoup accesses the HTML document and shows us the underlying structure of the page. In this example, we take a closer look at scraping pages from the online encyclopedia Wikipedia (Wikipedia 2016). The following is a small yet powerful script that demonstrates the use of BeautifulSoup (Example 6.4). The script uses BeautifulSoup (bs4) as well as the urllib module (import statement, Lines 1 and 2). The urllib

68  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

module provides a high-level interface for fetching Internet resources. In this example we fetch the “3D printing” page from Wikipedia (Line 4). The text (“response”) is then fed to BeautifulSoup. The prettify function then parses the tree into an easily understandable form, with nice indentations. This is then output to the screen (Line 6). Example 6.4. BeautifulSoup example from bs4 import BeautifulSoup importurllib.request response = urllib.request.urlopen(‘http:// en.wikipedia.org/wiki/3D_printing’) soup = BeautifulSoup(response) print(soup.prettify()) The results of this script are shown below (Output 6.6). Here, the first one or two dozen lines of the HTML document, which compose the page, are shown. The most important thing to note in the figure is that the HTML ­document is composed of many separate tags, which are wrapped in ­triangle brackets, that is, . Within each bracket, there are many separate attributes. Each of these can and must be accessed to get the useful information out of the web page. Searching for individual tags and getting the useful information out of the tags are discussed in the next example. Output 6.6. Scraped HTML sample



3D printing - Wikipedia, the free encyclopedia

window.RLQ = window.RLQ || []; window.RLQ.push( function () {

Parsing Tree-Structured Files  •  69

mw.config.set({“wgCanonicalNamespace”:””,”wgCanonicalSpecialPageName”:false,”wgNamespaceNumber”:0,”wgPageName”:”3D_printin Now the following block takes this HTML and extracts only the readable text (Example 6.5). The script begins similarly with imports. Then the page is fetched using the urllib.request module, fed to BeautifulSoup, and only the readable text is extracted from the web page. This results in a list of strings, contained in variable texts. There is a surprising amount of text in the HTML mark-up of a web page, and most of it is not intended for the reader! Example 6.5. Extracting readable text from HTML import re texts = soup.findAll(text=True) drop_list = [‘tr’,’table’,’sup’,’style’, ‘script’, ‘[document]’, ‘head’, ‘title’, ‘html’,’body’,’form’,’abbr’,’label’,’h3’,’h2’, ’h1’,’ul’,’div’,’blockquote’,’ol’] keep_list = [‘td’, ‘th’,’cite’,’i’,’strong’,’a’, ’b’,’span’,’li’,’p’] full_text = “” for element in texts:   name = element.parent.name   if name in drop_list:   #print(element.parent.name)   next  else:    el_list[name] = 0   if name in keep_list:    element = re.sub(‘\n’,’’,element)    element = re.sub(‘\s\s+’,’’,element)    n = len(element)    if (len(element) > 0):     full_text += “ “     full_text += element full_text = full_text.strip()

70  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

full_text= re.sub(“ print(full_text)

“, ‘ ‘,full_text)

This script checks the parent name for each element in the HTML document, and returns false if it is not intended for the final reader. This includes style guides, scripts, head and title sections, and document descriptors. The final lines of the script just prettify the text before outputting it into a block on the script. At this point, there is still a lot of extraneous white space left over from the formatting of the page. The partial output from the script is shown below (Output 6.7). The result is entirely correct, if not easily readable for human eye. Nonetheless, this output can be used for further archiving or detailed analysis. Both these uses will be described in later chapters. Output 6.7. Example readable text from HTML navigation search pad printing lenticular printing holography MakerBot Part of a series on the History of printing Woodblock printing 200 Movable type 1040 Printing press c. 1440 Etching c. 1515 Mezzotint 1642 Aquatint 1772 Lithography 1796 Chromolithography 1837 Rotary press 1843 Hectograph 1869 Offset printing 1875 Hot metal typesetting 1884 Mimeograph 1886 Photostat and Rectigraph 1907 Screen printing 1910 Spirit duplicator 1923 Xerography 1938 Phototypesetting 1949 Inkjet printing 1951 Dye-sublimation 1957 Dot matrix printing 1968 Laser printing 1969 Thermal printing c. 1972 3D printing 1984 Digital printing 1993 3D printing, also known as additive manufacturing (AM), refers to various processes used to synthesize a three-dimensional object. [1] In 3D printing, successive layers of material are formed under computer control to create an object. [2] These objects can be of almost any shape or geometry, and are pr There is much more that can be accomplished with the BeautifulSoup library. In fact, a useful further extension involves building web crawlers. BeautifulSoup helps here by scrapping all the web links found on a page. We use these links to redirect the bot to the next step along its crawl.

Parsing Tree-Structured Files  •  71

6.3 MINING CONTENT FROM PDF FILES In this section we turn to reading full text out of the pdf format. As you may know, the pdf format is a commonly used standard for sharing documents across multiple platforms. Our library of choice for reading pdf format is pdfminer3k. This library ports an older library, pdfminer, and updates it to work with Python 3 syntax. As usual with external libraries, this needs to be installed on to your Python build using easy_install or an equivalent. The following script assumes you have already selected a pdf file with an interesting content and stored it in a local directory. Here, we’re using a Windows system and have stored the pdf file in our data directory. The particular pdf file has an interesting content in the field of nanotechnology. This particular article, published in Chemical Review, is among the highest cited articles in the entire field of nanotechnology. pdf documents are collections of graphical objects to be laid out on a page. As a result, it takes a comprehensive and object-oriented design to try and make the best of the sometimes ambiguous structures found within the pdf file. pdf parsing is also time and memory consuming. Parsing is often done in a lazy fashion, parsing only what is needed and when it is needed (Shinyama 2016). In the example below, you will see a variety of different objects intended to facilitate the intelligent parsing of a pdf file. Much of the work is intelligently hidden from the user within these objects and therefore, the objects themselves do not warrant much discussion. The first few lines in the script below provide the import statements. A file handle, which lists the pdf file by name and directory, is shown here. We then create the parser, an empty document, and assign the document to the parser. Further instructions set the layout parameters, a resource manager, and a device. The apparent complexity of setting this up stems from the fact that the pdf format builds off of postscript files, which in turn provide instructions for printers for the typesetting and layout of documents. The particular object-oriented design of the pdfminer package results in a higher code reuse. This script culminates in an interpreter, which is used in processing the page (Example 6.6). Example 6.6. Example use of PDFMiner frompdfminer.pdfparser import PDFParser, PDFDocument frompdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter frompdfminer.converter import PDFPageAggregator

72  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

frompdfminer.layout LTTextLine

import LAParams, LTTextBox,

fp = open(‘Data/Nano/Tree Format/Daniel2006.pdf’, ‘rb’) parser = PDFParser(fp) doc = PDFDocument() parser.set_document(doc) doc.set_parser(parser) doc.initialize(‘’) rsrcmgr = PDFResourceManager() laparams = LAParams() device = PDFPageAggregator(rsrcmgr, laparams = laparams) interpreter = PDFPageInterpreter(rsrcmgr, device) doc_string = “” # Process each page contained in the document for page in doc.get_pages():  interpreter.process_page(page)   layout = device.get_result()   forlt_obj in layout:    ifisinstance(lt_obj, LTTextBox) or isinstance(lt_obj, LTTextLine):    newpage = lt_obj.get_text()     doc_string = doc_string+newpage print(doc_string) We now get each of the pages from the pdf, and process each page separately. These are passed onwards to an interpreter and the layout manager. For each object parsed on the page, we detect whether it contains text intended for the end user. If so, the object is rendered as text and appended to a string. The partial results of this script are shown below (Output 6.8). Output 6.8. Sample PDF output to text Chem. Rev. 2004, 104, 293-346 293

Parsing Tree-Structured Files  •  73

Gold Nanoparticles: Assembly, Supramolecular Chemistry, Quantum-Size-Related Properties, and Applications toward Biology, Catalysis, and Nanotechnology Molecular Nanosciences and Catalysis Group, LCOO, UMR CNRS No. 5802, Université Bordeaux I, 33405 TalenceCedex, France Marie-Christine Daniel and Didier Astruc* Contents 1. Historic Introduction 2. General Background: Quantum Size Effect and Single-Electron Transitions 3. Synthesis and Assembly 3.1. Citrate Reduction 3.2. The Brust-Schiffrin Method: Two-Phase The pdf is rendered prettily to text, including line breaks and spaces for new paragraphs. In reality however, the pdf format is optimized for the reader, and is not a handy format to work with tech mining. If you can, you should try to capture the data before it is encapsulated in pdf format. Nonetheless, when investigating full text sources, it might be your only option. As an example of some of the complexities encountered when reading pdf format, this particular document generated a nonfatal error because not all the objects on the page were found and rendered. Fortunately no text was lost. Other complexities include encryption or digital rights management. This may potentially limit your access to ready or copy your pdf files, even if you have the rights to access these files for reading. Some pdfs, but not all, have outlines that can be fetched with a get_outlines() method (Example 6.7). Example 6.7. Get outlines method outlines = doc.get_outlines() for (level,title,dest,a,se) in outlines:   print (level, title)

74  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

In practice you will want to scan whole directories of pdfs at once. This enables you to automatically mine multiple pdf files, for the purposes of comparing, structuring, and analyzing the contents. A more elaborate example that finds and reads files out of directories will be discussed in later chapters. While concluding the chapter, it is appropriate to consider what has been learned in the chapter and how it can be applied to text mining. Although the main focus of the chapter lies in parsing tree-structured files, the topic opens up capacities to mine a wider variety of sources for science and technology sources. The techniques discussed in the chapter would be very appropriate for mining the grey literature. Unlike publications or patents, grey literature has not been formally published. Nonetheless, substantial amounts of technology intelligence can be found in the grey literature of any organization. Publications and patents are only the tip of the iceberg in terms of assessing the knowledge, routines, and capabilities of any organization. Repositories of grey literature may include knowledge bases, intranet, or organizational web pages. These sources are particularly important when working with organizations without a formal research and development function. In this chapter then, we are rounding out a discussion of parsing relevant information from science and technology sources. We can look beyond formal publications, such as publications, and move on to other sources such as patents. Web pages, pdf documents, and news sources, they all help in assessing the full lifecycle of technology. Rounding out our processing capabilities also requires that we access and incorporate new Python libraries to our recipes. This parsed data can be followed with several extraction steps. During each of these steps just the information you need is extracted to produce informative lists and tables. This is a better and more productive use of your time at tech mining than a comprehensive scanning and cleaning of the data, which is performed all once. This extraction strategy, resulting in informative lists and tables, is the major topic of the next chapter.

CHAPTER 7

Extracting and Reporting on Text In the previous chapters, we described the process of gathering and parsing data. The newly structured data is saved in a JSON format for further use. Creating a JSON file is a robust and intermediate step in text mining. Creating such a file enables us to restart, reuse, and further expand our initial investigations into the data. In this chapter, we are now ready to use the data to produce informative lists and tables about our data. Whereas in Chapters 4, 5, and 6 we put data into structures, our goal in this chapter is to take it back out of the structures. As a result we pick up the discussion of iterating through data structures, first introduced in Chapter 3. We also provide some practical tools for managing the JSON archives, including splitting the archives according to features in the data. In these examples, we simplify some of the text processing methods to the author address and data published fields. This enables us to focus more closely on the reporting aspects of the problem in this chapter. An accompanying book devotes a whole chapter on the tasks of text and natural language processing. In the second half of the chapter, we describe working with basic data structures, both counters and dictionaries, to create interesting outputs. We provide a number of recipes for producing very useful, yet very generalizable, lists on our data. The true value of creating dictionaries from our data becomes apparent in the next chapter, when we use the dictionary to create full tables of data, using the Pandas Python library. In the next three sections, we begin a discussion on some useful information products to be gained from mining text. We’ll be using text information to answer a series of journalist’s questions—who, what, when, where, and why—about the data. In this chapter these questions are

76  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

p­ artially addressed using top 10 reports, created using the counter object that is first discussed in Chapter 3. We also discuss using dictionaries to build up the whole tables of data. This is primarily discussed in Chapter 8 of the book. In Chapter 1 of the book we described a simple approach, based on the “journalist’s questions” (Table 7.1). These questions can be used to appraise the kinds of technological intelligence you can create using your data. Now that we’ve begun a discussion about extracting and reporting the data, it is time to revisit these questions. Information products come in successive levels of complexity. Simple lists, such as a top 10 author list or a top 10 publishing organization list, give quick insight into the data. Tables provide comprehensive input suitable for further analysis. For instance, you can create a set of articles fully indexed by content. Then it becomes possible to filter and retrieve your content. This often reveals surprising relationships in the data. You can also compile organizational collaborations across the articles. Like article indices, these collaboration tables are often inputs to data analysis or machine learning routines. The final form of information products, we’ll consider here, is crosstabs. Cross-tabs mix two or more of the journalist’s questions to p­ rovide more complex insight into a question of research and development ­management. For instance, a cross-tab, which shows which organizations specialize in what content, can be used for organizational profiling. A decision-maker may use this as an input into the questions of strategic alliance. The variety of information products that we will be considering in Chapters 7 and 8 are shown in Table 7.2. The table shows the type of question being asked, as well as the form of the information product. We ­haven’t been able to describe every possible information product you might create in this table, but we have provided a representative sample with many code examples to follow.

Table 7.1.  Sourcing the journalist’s questions Type of question Who What When Where Why

Available fields Author Output, Subject Category, Keyword, Content Year Organization, Nation, Nation Filed, City Citations Made, Citations Received

Extracting and Reporting on Text  •  77

Table 7.2.  Information products Type of question Who What

When Where

Lists Most frequent authors Most frequent words

Most frequent years Most frequent nations Most frequent ­organization

Tables

Indexed articles

Cross-tabs

Cross-tabs of content by year Cross-tabs of content by ­organization

Articles by year Articles by organizations and collaborators

Cross-tabs of organization by year

Why

7.1  SPLITTING JSONS ON AN ATTRIBUTE In previous chapters we parsed the text data, storing it in data structures for further reuse. These data structures are then serialized for later reuse. Here we pick this backup again by loading from JSON into a data structure in memory, and iterating through each of the records. While iterating it is easy to examine specific criteria of interest across the records. Using this information about the data, we can easily split the data into two different sets. Suppose for instance we wanted one set of ­articles ­published before the year 2010, and another published ­thereafter. The ­following code provides an example of how this might be done (Example 7.1). Example 7.1. Splitting a corpus new_rec = {} old_rec = {} ​ forart_idin data:   record = data[art_id]   date = record[‘DP’]   year = int(date[0:4])

78  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

  if (year >= 2010):    new_rec[art_id]=record  else:    old_rec[art_id]=record ​ print(“In the newer set there are”,len(new_rec), “articles”) print(“In the older set there are”,len(old_rec), “articles”) The results are as follows (Output 7.1). Output 7.1. Results from splitting In the newer set there are 252 articles In the older set there are 247 articles It is often the case that we will want to split or merge our corpus. There could be any number of different criteria for splitting the data; splitting by year is just one possible use case.

7.2 MAKING A COUNTER Now that we’ve clarified how to get the data out of the JSON, it is time to produce some informative products based on the data. The simplest and perhaps the most useful product are lists, based on the counter object. In the following code block we find and extract the year of publication for the article (Example 7.2). The objective is to get an idea about the years that are predominant in the data. Example 7.2. Making a counter import collections as c year_counter = c.Counter() forart_idin data:   record = data[art_id]   date = record[‘DP’]   year = date[0:4]   year_counter[year] += 1 print(year_counter)

Extracting and Reporting on Text  •  79

Here is what the counter looks like when it is printed out in raw format (Output 7.2). The counter also provides some useful reporting features, which are described in the following section. Output 7.2. Screen output from a counter Counter({‘2013’: 51, ‘2014’: 50, ‘2015’: 39, ‘2011’: 38, ‘2012’: 38, ‘2010’: 36, ‘2009’: 35, ‘2008’: 34, ‘2006’: 20, ‘2004’: 20, ‘2007’: 20, ‘2003’: 17, ‘2005’: 16, ‘2002’: 12, ‘1999’: 11, ‘2000’: 8, ‘2001’: 8, ‘1996’: 7, ‘1998’: 7, ‘1992’: 6, ‘1997’: 5, ‘1991’: 4, ‘1990’: 3, ‘1993’: 2, ‘1995’: 2, ‘1983’: 2, ‘1989’: 2, ‘1976’: 1, ‘1994’: 1, ‘1986’: 1, ‘1974’: 1, ‘1988’: 1, ‘1985’: 1})

7.3 MAKING SIMPLE REPORTS FROM THE DATA In this section, we continue our example of making simple reports from the data. These reports aggregate common elements from our text database, and count their frequency in the corpus. We’ll discuss three related examples—one involving reports on years, one involving reports on authors, and one involving reports on countries. The following code example picks up on the year counts of the previous section by demonstrating a useful function of the counter object. In the following code block, we use the “most_common” function of the counter. In the parentheses is the number of most common elements we want to display (Example 7.3). Example 7.3. The most common method year_counter.most_common(10) This block gives the following results (Output 7.3). Output 7.3. The top 10 years [(‘2013’, (‘2014’, (‘2015’, (‘2012’, (‘2011’,

51), 50), 39), 38), 38),

80  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

(‘2010’, (‘2009’, (‘2008’, (‘2004’, (‘2007’,

36), 35), 34), 20), 20)]

As can be seen, the most frequent year in the data is 2013, with 2014 coming close behind. This top 10 list may not actually fully suit our ­purposes, but we will pick this example backup, both with dictionaries and with tables in the next chapter. Let’s now take an example where we examine the most frequent authors in a corpus and produce a simple report (Example 7.4). Here again we create a counter, we go through each article in the corpus, and we fractionally increment the credit for the author. This depends on how many other authors have also contributed to the article. Example 7.4. Counting authors author_counter = c.Counter() for article_id in corpus:   record = corpus[article_id]   if (‘AU’ in record):    author_list = record[‘AU’]    for author in author_list:     author_counter[author] += 1/len(author_list) author_counter.most_common(10) The results of this code block are shown as follows (Output 7.4). Output 7.4. Top 10 authors [(‘Langer R’, 4.733766233766234), (‘Wang Y’, 4.7254329004329), (‘Wang J’, 3.828787878787879), (‘Prausnitz MR’, 3.6666666666666665), (‘Li J’, 3.3730519480519487), (‘Zhang L’, 3.319877344877345), (‘Zhang Y’, 3.295310245310246), (‘Zhang X’, 2.6972582972582977), (‘Liu Y’, 2.6944805194805195), (‘Li Y’, 2.629401154401154)]

Extracting and Reporting on Text  •  81

The most frequent author is Langer. We must be cautious in interpreting these results because this database does not disambiguate authors. For instance, there could be many authors with the first initial “L” and a family name “Zhang.” We could perhaps attempt to do this by hand using additional information about the topic, or organizational affiliation of the authors. Nonetheless this is sufficient for this simple example demonstrating how to create reports out of various fields in the database. In this final example of producing reports (Example 7.5), we read again through our structured corpus, loaded from a JSON file. Here we look up the countries from the record, which takes the form of a list, and we increment the count for each of these countries to a counter. In this particular example we are going to fractionally weight each country. This is because there may be many contributing countries to a given article, and we only want one article to count once, regardless of the number of authors from that country. As an example, if there are seven contributing countries to an article, the length of the country list is seven. Each country on the list receives one seventh of the total credit. If all countries are China, for instance, China would receive full credit for a single article. Example 7.5. Counting nations new_country_counter = c.Counter() for article_id in corpus:   record = corpus[article_id]   country_list = record[‘AD1’]   for country in country_list:    new_country_counter[country] += 1/ len(country_list) We can now report on the most frequent countries in this field of research by taking the counter, and using the most_common() function. The results of this call are as follows (Output 7.5). Output 7.5. Top 10 nations [(‘USA’, 657.7502192252207), (‘China’, 231.00484367054105), (‘UK’, 137.92499999999993), (‘Japan’, 119.95833333333333), (‘India’, 115.61428571428576),

82  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

(‘Germany’, 106.36091686091689), (‘Italy’, 67.27777777777783), (‘Republic of Korea’, 54.437846068659994), (‘France’, 51.941666666666684), (‘Australia’, 50.6031746031746)] So in this field of research, the United States is the most productive country, while China and the United Kingdom come at number 2 and 3, respectively. One element of reporting we haven’t yet paid much attention to is the “why” question. Why questions in scientific writing are often indicated by the use of citations. Forward citations are made to other relevant work. Forward citations are the means by which a scientific author describes their chosen field, carves out their niche, and stakes their contribution (Swales 1990). Total earned cites by a paper represent backward cites, a partial indicator of the importance or preeminence of the specific paper under examination (Martin and Irvine 1983). Citations also play an important role in patent analysis. Citations are acknowledgment of prior art. In acknowledging prior art, the inventor stakes a claim to the uniqueness of their invention, and he or she is given temporary rights to exclude others from producing the invention. In return, the assignee contributes the patent to the domain of public knowledge. Hyperlinks are also an important element of answering “why.” These can be found on web or wiki entries. The collective forward or backward linkage of hyperlinks demonstrates hubs and authorities in an online environment (Kleinberg 1999). So the upshot of this is that citations, hyperlinks, and acknowledgments of prior art are a very valuable source of technology intelligence. Using the examples given in this chapter, you can easily create top 10 lists of the most significant links in your corpus. Expanding to the materials covered in Chapter 8, you can also produce tables of the most cited articles, and create cross-tabs revealing cocitation patterns.

7.4 MAKING DICTIONARIES OF THE DATA The objective of this final section is to create a simple association between the article and one of its attributes. Here we associate the article by its year of publication (Example 7.6). We might want such a simplified structure in order to create tables, which are discussed more fully in Chapter 8. Or we might want to do this to get a better insight into the data by sorting, ranking or alphabetizing the data.

Extracting and Reporting on Text  •  83

Example 7.6. Extracting a dictionary year_dict = {} for article_id in corpus:   record = corpus[article_id]   if (‘DP1’ in record):    year = record[‘DP1’]    if (year in year_dict):     year_dict[year] += 1   else:     year_dict[year] = 1 Other possible associations, useful in structuring a full table, could be by the total number of citations for the article (for instance in the Web of Science database). Or you could make an association by the patent office that awarded a patent, in a patent database. These dictionaries are not yet informative, in and of themselves. However, they are important waypoints to creating full tables of data, which is discussed in the following chapter.

7.5  COUNTING WORDS IN DOCUMENTS We begin the next example by loading the preprocessed articles from a JSON corpus. This corpus is constructed using the methods discussed in Chapter 4. The resultant JSON is a dictionary of dictionaries. The outermost dictionaries contains each of the records, indexed by a unique a­ rticle id. The innermost dictionary contains each of the fields of the record, indexed by a field id that is unique to the record. The fields are of course repeatedly used across articles. Every article has a title (field “TI”), but there is only one title field per article. The following code block loads the JSON into the appropriate data structure, called “corpus.” As noted, this is a dictionary of dictionaries (Example 7.7). Example 7.7. Loading a JSON import json with open(‘Output/nano_json.txt’, ‘r’) as inputfile:   corpus=json.load(inputfile) When processing the corpus, we iterate through each article of the corpus, fetching the appropriate field (Example 7.8).

84  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

Example 7.8. Fetching a field import collections as c indexed_articles = {} for number, article_id in corpus:   index = c.Counter()   record = corpus[article_id]   content = record[‘CO’]   words = content.split(‘ ‘)   for word in words:    index[word]+=1   indexed_articles[article_id] = index In this chapter, we focus exclusively on tabulating the data. To do this, we create a counter object for each record. Then we store each word found in the record in the counter. This increments a count of every time the word is found and repeated. One counter for each record is then saved in the indexed_articles dictionary. A sample record counter looks like the following (Output 7.6). Output 7.6. Sample counter Counter({‘of’: 24, ‘the’: 22, ‘and’: 14, ‘a’: 8, ‘to’: 8, ‘ferrocene’: 6, ‘that’: 6, ‘graphene’: 5, ‘this’: 4, ‘properties’: 4, ‘oxide’: 4, ‘its’: 4, ‘redox’: 4, ‘in’: 3, ‘ionic’: 3, ‘reduced’: 3, ‘from’: 3, ‘for’: 3, ‘complexes’: 3, ‘atomic’: 2, ‘il’: 2, ‘subsequent’: 2, ‘structural’: 2, ‘is’: 2, ‘intercalated’: 2, ‘by’: 2, ‘functionalized’: 2, ‘fe(ii)’: 2, ‘rearrangement’: 2, ‘rgo’: 2, ‘materials’: 2, ‘organometallic’: 2, ‘results’: 2, ‘nanocatalyst’: 2, ‘synthesis’: 2, ‘graphitic’: 2, ‘or’: 2, ‘structure’: 2, ‘between’: 2, ‘rgo-il’: 2, ‘solvent’: 2, ‘environmental’: 2, ‘’: 1, ‘after’: 1, ‘directing’: 1, ‘broader’: 1, ‘covalent’: 1, ‘leads’: 1, ‘directs’: 1, ‘activity’: 1, ‘pair’: 1, ‘anionic’: 1, ‘active’: 1, ‘exploiting’: 1, ‘emphasize’: 1, ‘non-toxic’: 1, ‘study’: 1, ‘was’: 1, ‘synthetic’: 1, ‘transfer’: 1, ‘behavioral’: 1, ‘transition’: 1, ‘increases’: 1, ‘characteristics’: 1, ‘fe’: 1, ‘hydrolysis’:

Extracting and Reporting on Text  •  85

1, ‘best’: 1, ‘use’: 1, ‘work’: 1, ‘useful’: 1, ‘functionalities’: 1, ‘approach’: 1, ‘bonds’: 1, ‘processes’: 1, ‘application’: 1, ‘reorganization’: 1, ‘diversify’: 1, ‘non-covalent’: 1, ‘used’: 1, ‘growing’: 1, ‘self-assembly’: 1, ‘core’: 1, ‘corresponding’: 1, ‘based’: 1, ‘at’: 1, ‘electron-transfer’: 1, ‘engage’: 1, ‘cr(vi)’: 1, ‘knowledge’: 1, ‘interactions’: 1, ‘consistent’: 1, ‘reversibly’: 1, ‘liquid-induced’: 1, ‘matrix’: 1, ‘potentially’: 1, ‘generation’: 1, ‘be’: 1, ‘area’: 1, ‘solvated’: 1, ‘represent’: 1, ‘ionothermal’: 1, ‘resulting’: 1, ‘glycerol’: 1, ‘[bmim][fecl4]’: 1, ‘driven’: 1, ‘general’: 1, ‘combined’: 1, ‘generate’: 1, ‘regional’: 1, This counter contains a mix of common and uncommon words. The common words, known formally as “stop words,” include some of the most common words in the English language, including “and,” “of,” and “the.” Many analysts remove or down-weight these words as they are seemingly obligatory parts of writing in English. They give very little specific indication of the content of the article. On the other hand, the counter also reveals a number of distinctive scientific terms including “nanoparticles,” “nanocatalytic,” and “hexavalent.” Words such as these, collected in parallel across thousands of articles, may be strongly indicative of new trends or developments in the field. Indices such as these are necessary inputs for further analysis. First however, we need to get this article index into a data frame where it can be further stored or processed. In the previous example, we counted the words unique to each record. Let’s now modify the script to take a look at the most frequent words in the corpus as a whole (Example 7.9). In the following script, we create a counter, total_words, and count all the words in the entirety of the corpus. Notice however the list of words we drop. As noted earlier, this is a partial stop word list of words that are commonly ignored when assessing content. Example 7.9. Counting a field import collections as c total_words = c.Counter()

86  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

ignore = [‘a’,’the’,’of’,’and’,’in’,’to’,’with’, ’for’,’by’,’was’, ‘on’,’were’,’as’,’is’,’that’, ’at’,’this’,\  ‘from’,’are’,’using’,’an’,’be’,’from’, ‘which’,’high’,’all’,’we’,’method’,’results’] fornumber,article_id in enumerate(corpus):   record = corpus[article_id]   content = record[‘CO’]   words = content.split(‘ ‘)   for word in words:    if (word not in ignore):     total_words[word]+=1 After running the script, we have the total_words counter. We can use this to find the top 10 most frequent words using the most common method. The following list is the output (Output 7.7). Output 7.7. The most frequent words [(‘nanoparticles’, 5469), (‘properties’, 3830), (‘surface’, 3098), (‘carbon’, 2687), (‘temperature’, 1954), (‘electron’, 1833), (‘oxide’, 1815), (‘synthesis’, 1753), (‘performance’, 1735), (‘structure’, 1727)] Despite the concerns that our database query might return a lot of “metric” nano words, and not words indicative of nanoscience and nanotechnology, the top 10 list looks pretty good. Let’s use this top 10 list to create a brief index to the data. As we close out the chapter, let’s return to the information products discussed at the top of the chapter. Table 7.3 lists a dozen kinds of information products in specific. We’ve covered many of the simpler products, which take the form of top 10 lists and reports in this chapter.

Extracting and Reporting on Text  •  87 Table 7.3.  Coverage of information products

Type of question Who

Form of product List

What

List

When

Example

Discussed

Most frequent authors Most frequent words

Section 7.3

List

Most frequent years

Section 7.3

Where

List

Most frequent nations, most frequent ­organization

Section 7.3 Section 8.2

Why

List

Citation analysis

Who What

Tables Tables

None given Indexed articles

Discussion in Section 7.3 Not discussed Section 8.1

When Where

Tables Tables

Why

Tables

Articles by year Articles by organizations and collaborators Citation analysis

Who What

Cross-tabs Cross-tabs

When Where

Cross-tabs Cross-tabs

Why

Cross-tabs

None given Cross-tabs of content by year Cross-tabs of content by ­organization None given Cross-tabs of organization by year Citation analysis

Section 8.1

Section 8.2 Section 8.2

Discussion in Section 7.3 Not discussed Section 8.3 Section 8.3

Not Discussed Section 8.3

Discussion in Section 7.3

88  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

In the chapter to follow, we will pick up on some of the more complex information products. In particular we will be looking more closely at the tables and cross-tabs discussed above. In the next chapter, we delve more deeply into producing tables or cross-tabs for the data. In particular, we introduce the Pandas package. Pandas is a “data frame” package, which gives us the capability of producing useful reports, summaries, and extracts of our data. This deepens the kinds of analyses that we have introduced this chapter.

CHAPTER 8

Indexing and Tabulating the Data The previous chapter described creating informative lists or reports on the data. This chapter describes a more elaborate process for tabulating data. The reason for doing this is three-fold. By assembling our data into a tabulated format, we can more easily interpret and comprehend the results. Second, the table format used enables us to build more complex queries than is otherwise possible with a simple top 10 list. And finally, these tables serve as input to statistics and machine-learning procedures. These techniques provide new insights into the data that could not otherwise be obtained without careful tabulation of the data. Our examples in this chapter are partial nanoscience and nanotechnology datasets drawn from the Web of Science database. Using this dataset, we produce more elaborate tables and cross-tabs. This continues the discussion about reporting begun in Chapter 7. As with the previous chapter, we have done some preprocessing of the text in this corpus. We’ve extracted all the content in the title, abstract, and keyword fields, and placed this in a custom field—“CC.” We’ve also done the same for the organizational names, placing this information in a dedicated field called “C2.” This data treatment deserves its own complete discussion. The accompanying book provides a complete chapter on text and natural language processing. The organization of the chapter is as follows. The previous chapter described extracting data into structures such as counters and dictionaries. This chapter describes pulling data into more complex structures including a dictionary of counters. These structures are useful for indexing the data so that we know which elements occur in which records. The first section of the chapter demonstrates how to index the data.

90  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

Section 8.2 introduces the Pandas package, which is our primary tool for tabulating the data. We demonstrate how to create dataframes from a variety of different data structures including lists, counters, and dictionaries. The product is readily interpretable tables of data that can be read in a spreadsheet-like format. We can even construct more elaborate tables by combining two or more tables through a “dot” or matrix multiplication procedure. This is discussed in Section 8.3. Section 8.4 describes some of the summary statistics available from the dataframe object in Pandas. In the following section, we pick up where we left off in Chapter 7. There we created a counter and discovered the most frequent words in our nanotechnology dataset. Now we use these most frequent words to create a partial index into the complete dataset. We will use this created index further in Chapter 8.

8.1  CREATING A PARTIAL INDEX OF THE DATA The example below begins by opening our JSON corpus of nanotechnology articles; the opening itself is not shown. The script itself begins with a list of the top interesting words found in the nanotechnology dataset (Example 8.1). Although this is a simplified example considering only 10 words, a more elaborate analysis with more words or phrases is possible. Example 8.1. Selective indexing keepers = [‘nanopartices’,’properties’,’surface’,’carbon’,’temperature’, \ ‘electron’,’oxide’, ‘synthesis’, ‘performance’, ‘structure’] brief_index = {} for number,article_id in enumerate(corpus): index = c.Counter() record = corpus[article_id] content = record[‘CO’] words = content.split(‘ ‘) for word in words: if word in keepers: index[word]+=1 brief_index[article_id] = index if (number < 10): print(index)

Indexing and Tabulating the Data  •  91

This example only indexes the word if it is on the keep list—our top 10 nanotechnology word list. Otherwise it runs the same as the previous indexing examples. The first 10 counters are output to screen, as shown below (Output 8.1). Output 8.1. Sample counters by records Counter({‘synthesis’: 3, ‘performance’: 2, ‘properties’: 1, ‘temperature’: 1}) ­ Counter({‘temperature’: 2}) Counter({‘properties’: 4, ‘structure’: 1, ‘surface’: 1, ‘electron’: 1}) ­ Counter({‘properties’: 6, ‘carbon’: 2, ‘performance’: 1, ‘electron’: 1}) ­ Counter({‘synthesis’: 1, ‘surface’: 1}) Counter({‘surface’: 3, ‘oxide’: 1}) Counter({‘electron’: 2, ‘properties’: 2, ‘temperature’: 1, ‘surface’: 1, ‘synthesis’: 1}) ­ Counter({‘carbon’: 3, ‘properties’: 1}) Counter({‘performance’: 4, ‘carbon’: 1}) Counter() The last article contains none of the top 10 words, and therefore the associated counter is completely empty. In this section we have created a fully indexed corpus of articles. This can be used for information retrieval, machine learning, or statistical analysis. We’ll discuss more about these topics in the following book. These examples also returned a shortened form of index, which can be used in producing cross-tabs or other forms of reports. Many databases utilize their own -curated categories of content. For instance, the Web of Science uses “subject categories.” These are also useful elements and aggregators of content, featured for instance in maps of science. These examples could easily be extended to include subject categories or other forms of curated content labels.

8.2 MAKING DATAFRAMES This section introduces the use of dataframes, by means of the Pandas package. The Pandas package provides useful features for the handling and displaying of data. Our purpose in this section is two-fold—both to introduce some essentials of Pandas programming, and also to develop some useful intermediate tables that can be used for further analysis.

92  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

The code example below shows how we can easily import our dictionaries into a dataframe object, thereby initializing a dataframe. In our example here, we pick up with a year dictionary. This will create a table that labels each article by the single year of its publication. For reasons of space we won’t go into detail on creating the year dictionary. You can find examples of doing this in Chapter 7, and also in the associated Python notebook for this chapter (Example 8.2). Example 8.2. Making a data frame from an index import pandas as pd import numpy as np dg = pd.DataFrame.from_dict(year_dict, orient=‘index’) dh = pd.DataFrame(index=dg.index) dg.astype(‘int’) dg.columns=[‘Year’] dg.head() The key method is the “from_dict()” command. We need to specify the orientation of the dataframe. The article IDs are on the rows, and so we oriented the matrix according to the index, or unique identifier for the table (Figure 8.1). Now this table can be useful for data visualization purposes. The year of publication could be useful supplementary information in a scatter plot or some other visualization. We’ll describe this in more detail in a ­follow-up chapter of the next book. However if you want to use this

Figure 8.1.  Dataframe of article ID by year.

Indexing and Tabulating the Data  •  93

year information as part of a more complex cross-tab with organizations, ­content, or some other field of data, you need to expand out the table. The code example below shows you how to create five new logical variables (Example 8.3). This expands the year field into five new variables, one for each of the years 2011 to 2015. Each of these variables contains a true or false variable. The variable is marked one for true and zero for false. Since the article is published in one and only one year, only one of the year variables can be marked as true. Example 8.3. Expanding the data frame dh[‘2011’] = np.where(dg[‘Year’]==‘2011’, ‘0’) dh[‘2012’] = np.where(dg[‘Year’]==‘2012’, ‘0’) dh[‘2013’] = np.where(dg[‘Year’]==‘2013’, ‘0’) dh[‘2014’] = np.where(dg[‘Year’]==‘2014’, ‘0’) dh[‘2015’] = np.where(dg[‘Year’]==‘2015’, ‘0’) dh=dh.astype(‘int’) dh.head()

‘1’, ‘1’, ‘1’, ‘1’, ‘1’,

The output of this code is shown in Figure 8.2. Let’s now examine the import of a fully indexed set of articles into Pandas. Pandas easily and gracefully imports more complicated data structures as well. The year example involves importing a dictionary; the following example involves importing a dictionary of counters ­(Example 8.4). We created the dictionary of counters in Section 8.1.

Figure 8.2.  Expanded dataframe by year.

94  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

Example 8.4. Using the head method import pandas as pd df = pd.DataFrame.from_dict(indexed_articles, orient=‘index’) df.head() This results are shown in Figure 8.3. There are a couple of additional finishing steps we will want to make to the table. First we can easily insert the missing values, here labelled with “NaN” or “not a number.” Second we need to specify the data type in the table. This will become important if we are to do further reporting or statistical analyses on the data (Example 8.5). Example 8.5. Filling missing values # fill in nans df.fillna(0, inplace=True) df=df.astype(int) This results are shown in Figure 8.4. This idiom of importing dictionaries of counters can be extended to analyzing other fields as well. In the following example, we extend this to finding all the organizations attributed to a paper, and creating a table of this information. The following code processes the custom field “C2” from the corpus. This field has been preprocessed to contain only the organization names.

Figure 8.3.  Dataframe of indexed data.

Figure 8.4.  Dataframe with filled missing data.

Indexing and Tabulating the Data  •  95

These are stored in a list data structure. Previous reporting has revealed the 10 most frequent organizations in the dataset. Let’s focus on these organizations in particular as we tabulate the data (Example 8.6). These are stored for processing in a list—inst_keepers. Example 8.6. Organizations of interest inst_keepers = [‘Nanyang Technol Univ’,’Zhejiang Univ’,’Jilin Univ’,’Univ Sci & Technol China’, \    ‘Shanghai Jiao Tong Univ’, ‘Seoul Natl Univ’, ‘Tianjin Univ’, ‘Beijing Univ Chem Technol’, \    ‘Huazhong Univ Sci & Technol’, ‘Univ Calif Berkeley’] We now progress in tabulating the data. Our goal is to take an extract from the corpus, storing the processed results in a dictionary of counters. This is readily imported into a Pandas dataframe, as demonstrated in the previous example (Example 8.7). Example 8.7. Selective organizational search brief_article_inst = {} for n,article_id in enumerate(corpus):   new_counter = c.Counter()   record = corpus[article_id]   inst_list = record[‘C2’]   if inst_list:    for inst in inst_list:     if inst in inst_keepers:      new_counter[inst] += 1/len(inst_list)    else:      new_counter[‘Other’] += 1/len(inst_list)  else:    new_counter[‘Empty’]=1   brief_article_inst[article_id]=new_counter As shown in the code, a little extra processing is needed before we can tabulate this information. The first step is to determine whether there is any list at all in the record. If not, we attribute the record as “Empty”

96  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

Figure 8.5.  Dataframe with organizations.

and move on. If we do not record the record as empty, the record will be dropped out of the dataframe. We do want to keep all the records to ease the cross-tabs, which will come later. If there is a list, we iterate through each organization in the list. If the organization is recognized, we credit that organization with one part of the whole record. This is proportional to the total number of organizations in the list. A single organization may get multiple credits from the total. If the organization is not recognized (and most will not be on the list), we credit that part of the record to “Other.” We now import the dictionary of counters, using the from_dict() method. We also fill in the missing values as above. The head() method displays just a few rows of the dataframe (Example 8.8). Example 8.8. Creating an organization data frame df_inst = pd.DataFrame.from_dict(brief_article_ inst, orient=‘index’) df_inst=df_inst.fillna(0) df_inst.head() The results are shown in Figure 8.5. There are 10 major nanotechnology organizations (many of them Chinese universities). As expected, there is also a field containing “None,” which indicates that a complete organizational address for all authors was not provided to this paper. The “Other” field, of which there are many in the database, indicates some other institutions than the top 10 contributed to this paper. In the next section we focus on creating cross-tabs based on these full tables.

8.3  CREATING CROSS-TABS Creating cross-tabs requires that we multiply, or dot, two of our dataframes together. In the examples that follow, we are always multiplying through the unique article id. The resultant cross-tab is therefore a summary or

Indexing and Tabulating the Data  •  97

aggregation of the data across multiple articles. The following code block demonstrates the process of shaping and transposing the data prior to multiplication (Example 8.9). Example 8.9. Sizing the data frame dft = df.T print(dft.shape) print(dh.shape) First we transpose the df dataframe. This dataframe contains the articles indexed by content. Our first goal is to produce a cross-tab that is dimensioned the number of words by the number of years. Let’s examine the dimensions of this transposed dataframe, and also the dh dataframe, which contains a table of years in an indexed format. These were described earlier in the chapter (Output 8.2). Output 8.2. The dimensions of the data frame (10, 5000) (5000, 5) The resultant tables are dimensioned 10 terms by 500 articles, and 5,000 articles by 5 years. If we multiply the resultant tables, we’ll get a single table dimensioned 10 by 5. The next example performs the multiplication, and assigns the result to the dataframe di (Example 8.10). Example 8.10. Creating a content by year cross-tab di= dft.dot(dh) di The resultant cross-tab shows the words of content on the rows, and the years of activity on the columns. The composite table therefore gives an indication of novel or trending content over time. There appears to be a sharp uptake in interest in the term “nanoparticles.” There is also an apparent rise in interest in the term “properties,” perhaps indicating a greater emphasis on material characterization. In contrast, the term “surface” seems to be in decline (Figure 8.6). A similar logic enables us to create a cross-tab of organizational activity over time. There is one new innovation in the code, and that lies in the formatting of the float variable. It is helpful to provide this in a standardized format without undue precision in the decimals (Example 8.11).

98  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

Figure 8.6.  Cross-tab of content by year.

Example 8.11. Creating an organization by year cross-tab df_inst_t = df_inst.T cross_inst_year = df_inst_t.dot(dh) pd.options.display.float_format = ‘{:,.2f}’.format cross_inst_year The resultant cross-tab shows the organizations on the rows and the years on the columns. The composite table therefore shows the pattern of activity by the organization over time. Although these are the most frequently publishing organizations, the organizations are a diverse lot. Even the most frequent publishers have fairly few articles in this sample. Perhaps most noteworthy is the uptick in missing address data in the year 2013 (Figure 8.7). In another code example we create a cross-tab of words by organizations (Example 8.12). There is another small innovation in this code example, and that lies in the drop command. Using this command, we can drop off uninteresting rows and columns. Since there is a lot of data to be displayed, let’s remove the “Other” and “Empty” fields. For that matter, let’s also drop out the “others” content term. Example 8.12. Creating a content by organization cross-tab cross_content_inst = dft.dot(df_inst) cross_content_inst.drop([‘Other’,’Empty’], axis=1, inplace=True)

Indexing and Tabulating the Data  •  99

Figure 8.7.  Cross-tab of organization by year.

Figure 8.8.  Cross-tab of content by organization.

cross_content_inst.drop([‘others’],axis=0,inplace=True) pd.options.display.float_format = ‘{:,.2f}’.format cross_content_inst The resultant cross-tab shows words on the rows, and organizations on the columns. Although there is only a rough measure of scientific specialization currently, this table shows clear differences across organizations. Some organizations, such as the University of Science and Technology of China, are seemingly strong across all major content terms. Others, such as the Beijing University of Chemical Technology, show a more concentrated profile of excellence. These three cross-tables are provided as examples, rather than complete analyses of trends in nanotechnology, and trends in organizational production of knowledge. A more complete analyses would choose organizations, years, and words to match the need of decision-makers. The

100  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

c­ orpus itself may need to be reworked and resampled. The results may also need to be nuanced by consultation with scientific experts. Nonetheless, we hope that it is clear that the methods used in this chapter could contribute significantly to such a reporting effort.

8.4  REPORTING ON DATAFRAMES In this final section of the chapter, we highlight a few useful dataframe methods, which can help in the summarizing and reporting of data. We will use our organizational table as an example. We’ve already noted the utility of the head() method. Sample output from this was shown earlier in the chapter as Figure 8.5. We’ve also described use of the shape attribute, which provides the row and column size of the table. Similar to shape is the info() method, which provides some technical information about the dataframe and its storage. The following is representative output from the method (Output 8.3). Output 8.3. The info method

Index: 5000 entries, WOS:000209009200006 to article_id Data columns (total 12 columns): Other 5000 non-null float64 Empty 5000 non-null float64 Nanyang Technol Univ 5000 non-null float64 Huazhong Univ Sci & Technol 5000 non-null float64 Univ Sci & Technol China 5000 non-null float64 Seoul Natl Univ 5000 non-null float64 Tianjin Univ 5000 non-null float64 Beijing Univ Chem Technol 5000 non-null float64 Jilin Univ 5000 non-null float64 Shanghai Jiao Tong Univ 5000 non-null float64 Zhejiang Univ 5000 non-null float64 Univ Calif Berkeley 5000 non-null float64 dtypes: float64(12) memory usage: 507.9+ KB Another useful method is the sum() method, which sums the rows or the columns of the dataframe. In this example, we will sum across rows resulting in a column total (Example 8.13).

Indexing and Tabulating the Data  •  101

Example 8.13. Summing a data frame df_inst.sum(axis=‘index’) The code results in the following summary (Output 8.4). Output 8.4. A summed data frame Other 4,178.79 Empty 585.00 Huazhong Univ Sci & Technol 18.92 Shanghai Jiao Tong Univ 24.16 Tianjin Univ 20.83 Nanyang Technol Univ 31.37 Univ Sci & Technol China 25.72 Zhejiang Univ 28.97 Beijing Univ Chem Technol 20.17 Jilin Univ 27.35 Seoul Natl Univ 20.83 Univ Calif Berkeley 18.91 dtype: float64 The final dataframe reporting feature that we would like to discuss is the describe() method. This provides useful summary statistics about the dataframe. The method call is shown below (Example 8.14). Example 8.14. The describe method df_inst.describe() The output of this is a new dataframe, as illustrated in Figure 8.9. This dataframe provides the count, mean, and standard deviation. It also gives the percentiles of the distribution. This is a good initial indication of the distribution of the data.

Figure 8.9.  The info method.

102  •  ANALYTICS AND TECH MINING FOR ENGINEERING MANAGERS

You may want to save the dataframe, for instance to import to other statistical packages. The following code block demonstrates how to do this (Example 8.15). Example 8.15. Saving a data frame cross_content_inst.to_csv(“Output/Nano/cross_ content_inst”, sep=’,’) The example makes use of the to_csv() method. Simply put in the desired destination, and note the appropriate separator. Here we have a comma separated file, but you might use tab characters “\t” or some other separator. The final chapter of the book offers some conclusions to our discussion on text mining. Here we reflect on what has been accomplished in this book with regard to text mining, and we also look ahead to the potential for text analytics using natural language processing, statistics, machine learning, and data visualization. We also revisit the text mining and analytics workflow, considering the benefits of a modular design. Part of this discussion is how and if we complement Python with other text mining applications.

Conclusions This concluding chapter offers some reflections on the book, and provides a look ahead to the companion book on text analytics. In the following sections, we’ll draw some general lessons or guidelines for text mining. This informs both the design of the book, and presents some specific workflow considerations to use when implementing tech mining. We’ll also conclude with a few thoughts about the correct balance of Python versus other tools for mining text.

HOW YOU TECH MINE Tech mining offers a new component to understanding science and technology data. With tech mining you can monitor using computers. This enables a broader scale and scope of monitoring than ever before. Tech mining also complements human expertise. The human scale of understanding is often deep and narrow. Scientists and engineers often understand their own fields extremely well, but have little or no oversight over surrounding fields of knowledge. In the words of Ziman (1987), scientists and engineers face the danger of “knowing everything about nothing.” With computer-aided techniques, we are increasingly able to survey vast fields of knowledge, built out of the collective thoughts and communications of millions of scientists, engineers, entrepreneurs, and enterprises. One example of a map of knowledge portrayed at a breath-taking scope is the work of Börner and colleagues (Börner et al. 2012). This map, beautifully rendered, shows the entirety of scientific knowledge as it is known today. This work assures us of the fundamental unity of human knowledge, and for many organizations, it enables organizations to place their contributions within a greater whole. Another work, which is similar in spirit and in methods, provides an equivalent view into the world of technologies and patents (Kay et al. 2014). Similar efforts, which map economic innovation, are also available (Hausmann et al. 2014). Computers transform, but they do not replace, human judgment. The collective decisions that animate these vast maps of science, ­technology,

104  •  CONCLUSIONS

and innovation remain somewhat obscure. Furthermore, there is an ever-present danger of assuming that the way the world is today is the way it must remain in the future. This of course is not true. The networks of knowledge, and the repositories of expertise, which exist in the world today will continue to grow and be transformed. It is important therefore to use tech-mining tools to enable decision-making about these critical topics. The computer processes that support tech mining have a lot in common with other data-mining and data-science processes. A match between the data and the organization is made. The match sometimes involves exploration, but it often entails much more directed questions such as forming strategic alliances, determining priorities, or evaluating existing competencies. Between finding the data and formulating recommendations comes intermediate steps where the data is collected, curated, analyzed, and displayed. This first book showed some of the promise of text mining for technology management. But in reality, we have only begun to discuss our existing, and potential future, capabilities for mining the vast repositories of human knowledge. In that light before we turn to a preview of the second book, let’s draw some lessons from using Python for text-mining purposes.

TEXT-MINING LESSONS At the beginning of the book, we argued for five injunctions of text mining. These injunctions, we argue, should guide a text-mining process and should inform processes or other analytical tools intended to support text mining. The injunctions we discussed are as follows. 1. 2. 3. 4. 5.

Introspection of the process is critical. The toughest problem in applied text mining is data transformation. Data structures are central to the text-mining process. Clear protocols for text inputs and outputs are essential. There must be clear stopping and starting points throughout the process.

Let’s explain and illustrate each of these injunctions in turn. These in turn provide design guidelines for selecting Python and implementing a text-mining script, or indeed any other text-mining software.

Conclusions  •  105

Introspection of the process is critical. In data mining, it is critical to get a feeling for your data. It perhaps is especially important with text mining because we’re so often dealing with a lengthy but repeated process. Rather than making a mistake and learning about it later, it is better to fail immediately, and thereby rapidly correct any error. There are two idioms that assist us in gaining insight into the data, and into any ongoing transformation. These are the print statement, and the enumerate statement. We introduce these first in Chapter 3, but make applied use of them throughout the book. Where we don’t put in such statements, you should add them yourself and take a look at any intermediate steps of the process. The toughest problem—at least in our context of applied text mining—is the process of data transformation. There is a certain irreducible complexity to scraping the data and reworking it into a format that can be understood, reported, and analyzed. Making a black box is no answer to the problem, because you can’t see inside the box, correct mistakes, or adapt it for new data sources. Managing the complexity requires some amount of overhead. We think this overhead in learning is best spent in finding a compact and powerful notation for transforming the data. We’ve confronted these issues of learning and complexity in writing Chapters 5 and 6. Data structures are central to the text-mining process. The central role of data structures is implicit in the whole need for appropriate data transformation. A data structure is the workhorse of any such transformation. The primary choice in utilizing a data structure is whether a single all-­ purpose structure can be made to fit, or whether we should use multiple structures as needed. In Chapters 7 and 8, we found it necessary to use lists, dictionaries, and even more specialized structures such as ­counters. These were even compounded into more elaborate structures such as ­dictionaries of counters. Clear protocols for text inputs and outputs is essential. The Java­Script Object Notation (JSON) file structure has proven central to this book, throughout the scripting chapters. Another important question here is whether we should create multiple outputs from the text-mining process, or a single corpus that is annotated with new information as necessary. In this book we’ve tended to stick to a single, comprehensive JSON file with custom fields for storing new information. There are other aspects to the problem as well, including a clear file directory structure to store text inputs and to store derivative outputs. There must be clear starting and stopping points through the process. Life is busy, scripting is fragile, and script reuse is often hard and

106  •  CONCLUSIONS

c­ onfusing. It is better to take small, short steps in the mining process, where each of the steps can be easily understood, easily interrupted, and easily modified. This argues against a large, standardized, and monolithic process. There are a number of rational processes available to guide data mining. This includes the popular cross-industry standard process for data mining (CRISP-DM) (Chapman et al. 1999). Such models have been extended to consider concerns specific to data science (Baumer 2015). Nonetheless, we feel the text-mining process is poorly captured by any such rational process. The five injunctions discussed above suggest we should be learning, teaching, and operating with a leaner, more agile ­process. Rapid prototyping processes are more appropriate than a rational process, even if the process contains many loops for feedback or iteration. We feel the iPython notebook is well-suited to support an agile process of text mining. The notebook runs interactively, permitting a lot of introspection of the code and intermediate steps. Notebooks also permit us to document and reproduce specific steps in any transformation ­process. Finally, the notebook imposes a kind of discipline of having fixed the starting and stopping points, with intermediate outputs. These virtues of the notebook environment are independent of, and build upon, the merits of Python itself. There are some items that we didn’t cover, but might have liked to have covered in detail. One topic concerns metadata—it would be nice to know the derivation of any specific data source in our text-mining output. That way we could clearly track our text-mining products, and distinguish source and derived data. Such data could be stored alongside our dictionaries of dictionaries, perhaps as a supplementary field, and then stored in JSON format. We’d also like to cover more about bots, crawlers, or agents for sourcing data. Despite the prevalence of such bots, most of them are created in practice. There is a lack of resources that are guiding the construction of bots for information sourcing and web scraping.

TEXT ANALYTICS We do however have still more to say on the topic of text analytics, the topic of the companion book. In this section we provide a brief overview. The primary topics of the book are text and natural language processing, machine learning, geographic mapping, visualization, and statistical inference.

Conclusions  •  107

The goal of text and natural language processing is to further clean and treat text data prior to analysis. The new book will revisit some details about string handling, including splitting, stripping, and handling lower and uppercase strings. Coverage of this chapter also includes regular expressions—a language for expressing and executing text patterns. We discuss both searching and substitution. The chapter also provides some information about natural language processing, care of the NLTK ­(“Natural Language Tool Kit”) package. We’ll be discussing handling stop words, and tokenizing our texts into words and phrases. The chapter also discusses using NLTK to identify parts of speech. The goal of machine learning is to impose simple, robust structures on the data so that we can learn and generalize from the data that has been collected. The heart of the book is the discussion of text analytics. Our choices here are to cover two workhorses of text analytics—factors and clusters. The original factor models include a range of linear state space models. We update these techniques with coverage of a new, stateof-the-art non-linear mapping technique called t-stochastic neighborhood embedding. The classic clustering techniques include link clustering and k-means modelling. These traditional clustering methods are updated with a new technique, the popular and versatile technique of topic modelling. A real strength of Python is that all these techniques are available native in Python, in the scikit learn (sklearn) package. Text visualization is as important to text analysis as it is to data analysis. These techniques enable us to explore, discover, and communicate our findings. We’ll cover some of the classic data visualization and techniques, as well as discussing new techniques particularly adapted to text. These new techniques include word clouds. Innovative networks operate across space, and organizational boundaries. Decision-makers are increasingly interested in the coproduction and emergence of new technologies in districts and in regions. Network analytic techniques help us to identify critical nodes (whether articles, ­concepts, people, or organizations), critical to the functioning of a network of knowledge. We’ll discuss network structures, layout, and metrics using the Networkx package. We’ll also provide tools for geo-locating organizations, and plotting them on geographic maps. This enables the creation of specialized maps by grid or by region. Statistical inference enables us to draw valid conclusions from our data. Text data is always sampled, always incomplete, so it is critical to draw reasonable inferences from the data given the uncertainty and incompleteness. Critical issues here involve questions of whether certain terms

108  •  CONCLUSIONS

are more frequent in one corpus than another, and whether one organization is publishing (or being cited) more than another. Many of the data outputs from this book are directly consumable by analysts and decision-makers. These include the cross-tabs and top 10 lists we discussed in Chapters 7 and 8. But the scripts and methods discussed in this book remain valid and necessary for the next book. Lists, tables, and cross-tabs are all necessary inputs to text analysis, or can be further enhanced by network or other visualization techniques. The geographic coverage of the new book adds a significant new dimension to our ability to analyze the “where” in technology management. Text mining remains a significant precursor to text analysis, and we will continue to build upon the materials introduced in this book.

USING THE SCRIPTS IN THIS BOOK We’ve provided many scripts and examples in this book; now is the time to reflect on their intended use. The notebooks can be taken as individual recipes for solving a specific problem. By sequencing and solving a number of different notebooks, we can perform a complete tech-mining analysis. More powerful and useful still is the idea that we can customize to existing notebooks to perform an even greater variety of tasks. A full process for performing a single, specific tech-mining study might look like Figure C.1. The goal of this representative tech mining study is to produce a technology map. Using such a map an organization could produce a whitespace analysis to determine new opportunities for innovation, and to better understand where partners and competitors sit within this landscape. This box and arrow diagram consists of two kinds of entities. One entity,

Metadata

Model base

2

6

Patents an a Tree-format

Parsing

1

3

Text and natural language processing 5

Tabulating and indexing

Factor analysis

Data visualization

7

8

9

4 Patents in a JSON corpus

Figure C.1.  Process for tech mining study.

Patent map for white-space analysis 10

Conclusions  •  109

r­ epresented by the file symbol, consists of either data or information products. The other, represented by a box, represents an iPython notebook. The study begins with patents in a tree-like format (Item 1). Patents are often downloadable in XML format, a tree-structured and self-documenting file format. These are then parsed (Item 3) and stored in a standardized JSON corpus (Item 4). Sometimes the patents require extra postprocessing in the text or language (Item 5). This is something we will cover in the accompanying book. The JSON corpus is then fully indexed (Item 7). The index can be further analyzed using factor analysis or other latent-space modeling techniques (Item 8). We then visualize the map, perhaps supplementing the map with additional information such as the date of the patent or the patent portfolio of specific assignees (Item 9). We then output a map in support of a patent white-space analysis (Item 10). Items 8 to 10 are discussed in the accompanying book, with supplementary examples. Two items that are not part of the best practices, but we could ­easily see becoming state-of-the-art in the future, are metadata (Item 2) and model bases (Item 6). Metadata could be used to describe the format of a standard record, the date at which the data was collected, and any special changes made to the data. This could help when handling the provenance of a wide-variety of data sources. A model base, perhaps built on top of the Pandas database, could simplify the information transfer between scripts. Such a base could be used to store and replicate specific models. It is also good practice to make a firewall between the derived and modelled data and the original source data. Now if the process described above was something that you needed to do regularly, you could standardize the scripts and put them into ­production. You may even want to automate the collection and sourcing of the data using bots. Standardizing the scripts means keeping the basic ­operation of the software, but adding in additional checks and error ­handling. Without an analyst in a loop, surprises happen often.

PYTHON AS PART OF THE TEXT-MINING ENVIRONMENT We feel that Python is a relatively complete solution for text mining. With the iPython notebook, it meets many of the requirements for an effective means of mining and later analyzing text. Nonetheless the practicality is key. In this section, we consider mixing Python with other available tools. This may entail using Python for just the mining parts of the task, and moving to other tools for analysis. Or this may entail using other tools for

110  •  CONCLUSIONS

mining, and then adopting Python for analysis. We also discuss the merits of a completely Python native solution. If you were to take the results of this book onwards to other analysis packages, you will have benefitted from a compact and powerful language for transforming your text data into a form better suited for analysis. In fact for some of the tasks in this book, such as web scraping or extracting text from pdfs, there may be no other tool better suited for text mining than Python. Transferring your results may be as simple as saving your pandas data frame into a comma-separated file. Most statistical tools can import such comma-separated files. A range of other tools present themselves for the data analysis portion of the task, including engineering-related tools such as Matlab, statistical packages such as SPSS or SAS, and specialized tools for network analysis and visualization such as Tableau, Gephi, or Pajek. And we would remiss if we do not acknowledge the importance and value of the R-language, a full-featured language for statistics and data science. An alternative would be to pick up Python only for the text analysis part of the job. If this is your preference, you will have chosen other tools to begin the text-mining process. The authors of this book have wide familiarity with VantagePoint software, and can recommend it as an alternative to Python. VantagePoint is also integrated with the Thomson database where it is known as the Thomson Data Analyzer. Other proprietary packages (sometimes specialized by domain) are offered by companies including IBM and MonkeyLearn. Using Python at the analysis phase enables you to take advantage of the many high quality data analysis and visualization packages native to Python. Our favored answer is native Python, for both the text-mining and the text-analysis phases of the project. This minimizes data transfer, and eases replicability and reuse of your scripts. We feel that Python is bestin-class for the data acquisition and transformation task. Although the learning curve is high, the language is powerful and succinct. Implementations of the newest analysis and visualization techniques are often more readily available in Python then elsewhere. However the diagnostic tools are somewhat limited when compared to standard statistical computing software. So in conclusion to the book, we hope you find Python a useful and compelling solution to text mining, and we hope that this book has helped in further advancing your learning. Please join us for the follow-up book where we take what has been learned in this book and apply it to a range of valuable and interesting analyses using machine learning and visualization techniques. We find the field of tech mining a fascinating and valuable addition to a toolbox for managing high technology enterprises.

References Arora, S., J. Youtie, P. Shapira, L. Gao, and T. Ma. 2013. “Entry Strategies in an Emerging Technology: A Pilot Web-Based Study of Graphene Firms.” Sciento­metrics 95, no. 3, pp. 1189–207. doi:10.1007/s11192-013-0950-7 Baumer, B. 2015. “A Data Science Course for Undergraduates.” arXiv.org. http:// arxiv.org/abs/1503.05570 (August 1, 2015). Börner, K., R. Klavans, M. Patek, A.M. Zoss, J.R. Biberstine, R.P. Light, V. ­Larivière, and K.W. Boyack. 2012. “Design and Update of a Classification System: The UCSD Map of Science.” PLOS One 7, no. 7, p. e39464. http:// journals.plos.org/plosone/article?id=10.1371/journal.pone.0039464 (August 1, 2015). Chapman, P., J. Clinton, R. Kerber, T. Khabaza, T. Reinartz, C. Shearer, and R. Wirth. 1999. “CRISP-DM 1.0.” ftp://ftp.software.ibm.com/software/analytics/spss/support/Modeler/Documentation/14/UserManual/CRISP-DM.pdf (January 1, 2016). Continuum Analytics. 2015. “Download Anaconda.” Anaconda. http://continuum. io/downloads#py34 (June 4, 2015). Enthought Scientific Computing. 2015. “Download Canopy.” https://store. enthought.com/downloads/#default (June 4, 2015). EPO (European Patent Office). 2016a. “Home.” www.epo.org/index.html ­(January 1, 2016). EPO. 2016b. “Nanotechnology.” www.epo.org/news-issues/issues/classification/ nanotechnology.html Eysenbach, G. 2011. “Can Tweets Predict Citations? Metrics of Social Impact: Correlation With Traditional Metrics of Scientific Impact.” Journal of ­Medical Internet Research 13, no. 4, p. e123. doi:10.2196/jmir.201 Github. 2016. “Where Software Is Built.” https://github.com/ (January 1, 2016). Gök, A., A. Waterworth, and P. Shapira. 2015. “Use of Web Mining in Studying Innovation.” Scientometrics 102, no. 1, pp. 653–71. doi:10.1007/s11192-0141434-0 Guo, Y., X. Zhou, A.L. Porter, and D.K.R. Robinson. 2014. “Tech Mining to Generate Indicators of Future National Technological Competitiveness: Nano-enhanced Drug Delivery NEDD in the US and China.” Technological Forecasting and Social Change 97, pp. 168–80. doi:10.1016/j.techfore.2014.02.026

112  •  REFERENCES Hall, N. 2014. “The Kardashian Index: A Measure of Discrepant Social Media Profile for Scientists.” Genome Biology 15, no. 7, pp. 424–26. doi:10.1186/ s13059-014-0424-0 Hausmann, R., C.A. Hidalgo, S. Bustos, M. Coscia, A. Simoes, and M.A. Yidrim. 2014. The Atlas of Economic Complexity. Cambridge, MA: The MIT Press. Hilbert, M. 2014. “What Is the Content of the World’s Technologically Mediated Information and Communication Capacity: How Much Text, Image, Audio and Video?” The Information Society: An International Journal 30, no. 2, pp. 127–43. doi:10.1080/01972243.2013.873748 iPython Interactive Computing. 2015. “The iPython Notebook.” http://ipython. org/notebook.html (June 4, 2015). Katz, J.S. 2006. “Indicators for Complex Innovation Systems.” Research Policy 35, pp. 893–909. doi:10.1016/j.respol.2006.03.007 Kay, L., N. Newman, J. Youtie, A.L. Porter, and I. Rafols. 2014. “Patent Overlay Mapping: Visualizing Technological Distance.” Journal of the American Society for Information Science and Technology 65, no. 12, pp. 2432–43. doi:10.1002/asi.23146 Kleinberg, J.M. 1998. “Authoritative Sources in a Hyperlinked Environment.” Proceedings of the ninth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 668–77. Kline, S.J. 1985. “Innovation Is Not a Linear Process.” Research Management 28, no. 4, pp. 36–45. Ma, J., and A.L. Porter. 2014. “Analyzing Patent Topical Information to Identify Technology Pathways and Potential Opportunities.” Scientometrics 102, no. 1, pp. 811–27. doi:10.1007/s11192-014-1392-6 MacKenzie, D.J., and B. Özler. 2011. “The Impact of Economics Blogs.” Social Science Research Network. http://papers.ssrn.com/sol3/papers.cfm?abstract_ id=1921739 (January 1, 2016). Martin, B.R., and J. Irvine. 1983. “Assessing Basic Research: Some Partial Indicators of Scientific Progress in Radio Astronomy.” Research Policy 12, no. 2, pp. 61–90. doi:10.1016/0048-7333(83)90005-7 Mohammadi, E., M. Thelwall, and K. Kousha. 2015. “Can Mendeley Bookmarks Reflect Readership? A Survey of User Motivations.” Journal of the Association for Information Science and Technology. http://onlinelibrary.wiley.com/ doi/10.1002/asi.23477/abstract (January 1, 2016). NCBI (National Center for Biotechnology Information). 2016. “PubMed.” www. ncbi.nlm.nih.gov/pubmed (January 1, 2016). Newman, N.C., A.L. Porter, D. Newman, C.C. Trumbach, and S.D. Bolan. 2014. “Comparing Methods to Extract Technical Content for Technological Intelligence.” Journal of Engineering and Technology Management 32, pp. 97–109. doi:10.1016/j.jengtecman.2013.09.001 Porter, A.L., and S.W. Cunningham. 2004. Tech Mining: Exploiting New Technologies for Competitive Advantage. New York: John Wiley and Sons. Porter, A.L., and S.W. Cunningham. 2012. Tech Mining: Exploiting New ­Technologies for Competitive Advantage. Chinese edition. Beijing: Tsinghua University Press.

References  •  113 Porter, A.L., S.W. Cunningham, and A. Sanz. 2015 “Advancing the Forecasting Innovation Pathways Approach: Hybrid and Electric Vehicles Case.” Inter­ national Journal of Technology Management 69, no. 3–4, pp. 275–300. Priem, J., D. Taraborelli, P. Groth, and C. Neylon. 2010. “Altmetrics: A ­Manifesto.” http://altmetrics.org/manifesto/ (December 29, 2015). Python Software Foundation. 2015a. “Python.” www.python.org (June 4, 2015). Python Software Foundation. 2015b. “The Python Tutorial.” https://docs.python. org/3/tutorial/inputoutput.html (June 7, 2015). Pythonxy. 2015. “Scientific-Oriented Python Distribution Based on Qt and ­Spyder.” http://python-xy.github.io/ (June 4, 2015). Shapira, P., J. Youtie, and S. Arora. 2012. “Early Patterns of Commercial Activity in Graphene.” Journal of Nanoparticle Research 14, no. 4, p. 1–11. Shinyama, Y. 2016. “PDFMiner.” www.unixuser.org/~euske/python/pdfminer/ index.html (January 3, 2016). Stackoverflow. 2015. “All Questions.” http://stackoverflow.com/questions/ (June 6, 2015). Swales, J.M. 1990. Genre Analysis. English in Academic Research Settings. ­Cambridge: Cambridge University Press. Thelwall, M., S. Haustein, V. Larivière, and C.R. Sugimoto. 2013. “Do Altmetrics Work? Twitter and Ten Other Social Web Services.” PLOS One 8, no. 5, p. e64841. doi:10.1371/journal.pone.0064841 VantagePoint. 2016. “Turn Information Into KNOWLEDGE.” www.thevantagepoint.com/ (January 1, 2016). Veltri, G.A. 2013. “Microblogging and Nanotweets: Nanotechnology on Twitter.” Public Understanding of Science 22, no. 7, pp. 832–49. doi:10.1177/0963662512463510 Watts, R.J., and A.L. Porter. 1997. “Innovation Forecasting.” Technological Forecasting and Social Change 56, no. 1, pp. 25–47. doi:10.1016/S00401625(97)00050-4 Wikipedia. 2016. “Wikipedia.” www.wikipedia.org/ (January 1, 2016). Zahedi, Z., R. Costas, and P. Wouters. 2014. “How Well Developed Are ­Altmetrics? A Cross-Disciplinary Analysis of the Presence of ‘Alternative ­Metrics’ in ­Scientific Publications.” Scientometrics 101, no. 2, pp. 1491–513. doi:10.1007/s11192-014-1264-0 Zhang, Y., X. Zhou, A.L. Porter, and J. Vicente Gomila. 2014. “How to Combine Term Clumping and Technology Roadmapping for Newly Emerging Science & Technology Competitive Intelligence: “Problem & Solution” Pattern based Semantic TRIZ Tool and Case Study.” Scientometrics 101, no. 2, pp. 1375–89. doi:10.1007/s11192-014-1262-2 Ziman, J. 1987. Knowing Everything About Nothing: Specialization and Change in Research Careers. Cambridge: Cambridge University Press.

Index A altmetrics, 42 Anaconda, 12 ARPANET, 1 arrays, 18 B BeautifulSoup library, 67 Web scraping, 67–70 blogs, 42 Boolean query, 40 Bouzas, V., 30 C Canopy, 12 citation forward, 82 measures, 43 cited record field (CR), 39 column-formatted data, 35 reading, 47–51 compound data structures, 30–32 Continuum Analytics page, 12 corpus, 32, 48, 52, 54, 83 counter, 29, 84–85 counter, making, 78–79 CRISP-DM. See cross-industry standard process for data mining cross-industry standard process for data mining (CRISP-DM), 106 cross-tabs, 6, 76 creating, 96–100

D data collecting and downloading, 34–41 formats, 35 mining, 5, 7 unstructured, 35 data directory, 21 data structures, 26–30, 105 compound, 30–32 counter, 29 data transformation, 105 databases, 38 DataFrames, 18, 91–96 reporting on, 100–102 Delicious, 34, 43 Derwent, 38 describe() method, 101 Designated states (DS), 40 development environment, 13–17 dictionaries, 27 corpus, 32 reading and parsing, 54–56 reading and printing JSON dictionary of, 56–59 dictionary, defined, 48 distribution, Python, 12 E Eclipse, 13. See also integrated development environment (IDE) enumerate() method, 26

116  •  Index

EPO. See European Patent Office European Patent Office (EPO), 44, 62 Extensible Mark-up Language (XML) file, 62 reading, 62–67 F field tag, 52 fields, 38. See also records forward citations, 82 G github, 9, 25 H hyperlink, 40, 82 I IDE. See integrated development environment innovation, 33 integrated development environment (IDE), 13 International Patent Classification (IPC), 40, 44 invention, 33 IPC. See International Patent Classification iPython Notebook, 10, 13–14, 21, 22 J Java Script Object Notation (JSON), 24, 47, 105 reading and printing dictionary of, 56–59 splitting on attribute, 77–78 JSON. See Java Script Object Notation L library, 17 lists, 26

M machine learning, 107 machine-accessible format, 35 Matplotlib, 18 MEDLINE database, 44 Mendeley, 34 module, 17 N nanometer, 44 nanotechnology, 36, 44 National Library of Medicine, 40 Natural Language Tool Kit (NLTK) package, 107 NetworkX, 18 Networkx package, 107 NLTK. See Natural Language Tool Kit NumPy package, 18 O open source software, 1 output directory, 21 P package, 17–19 NetworkX, 18 NumPy, 18 Pandas, 18 Pip, 18 Scikit-learn, 18 SciPy, 18 Pandas package, 18 dataframes, 91, 95 parsers pdf files, 71 PubMed, 53–54 partial index, 90–91 pdf files mining content, 71–74 parsing, 71 pdfminer3k, 71 Pip package, 18 PMID, 52

Index  •   117

PubMed, 40, 44, 47 parsing, 53–54 Python, 2–3 coding, 3, 11 data structures, 26–30 dictionaries, 27 different version of, 11–12 distribution, 12 installing, 12–13 language, 19 lists, 26 meaning, 11 part of text mining, 109–110 scripts coding, 13 versions of, 11–12 Python Application Programming Interface, 19 R R language, 11 records, 38. See also databases row-formatted data, 35 reading, 51–53 S sample record counter, 84 Scikit-learn package, 18 SciPy package, 18 script, 17, 108–109 Python coding, 13 simple reports, 79–82 stop words, 85 syndicated database, 44 T t-stochastic neighborhood embedding, 107 tech mining, 103 lessons, 104–106

technology, 33 text analytics, 106–108 text mining, 3 input organization, 21 outputs organization, 21 python part of, 109–110 text visualization, 107 Text-Mining-Repository, 9 Thomson Data Analyzer, 110 to_csv() method, 102 tree-formatted data, 35 Twitter, 18, 34, 43 U University of Science and Technology of China, 99 unstructured data, 35 urllib module, 67–68 V VantagePoint software, 110 W walk method, 55 Web of Science, 35 web pages, 42 Web scraping, 67 BeautifulSoup library, 67–70 Wikipedia, 34, 43 wildcards, 35. See also Web of Science words, counting, 83–88 X XML. See Extensible Mark-up Language Z zip method, 50

OTHER TITLES IN OUR ENGINEERING MANAGEMENT COLLECTION C.M. Chang, Editor Business Fundamentals for Engineering Managers by Carl M. Chang An Introduction to Engineering Economics by Kal Sharma Practical Applications of Engineering Economics by Kal Sharma Manufacturing Excellence: Renewal and Rebirth by Kenneth N. Mckay and Vincent C.S. Wiers Lean Six Sigma and Statistical Tools for Engineers and Engineering Managers by Wei Zhan and Xuru Ding Momentum Press is one of the leading book publishers in the field of engineering, mathematics, health, and applied sciences. Momentum Press offers over 30 collections, including Aerospace, Biomedical, Civil, Environmental, Nanomaterials, Geotechnical, and many others. Momentum Press is actively seeking collection editors as well as authors. For more information about becoming an MP author or collection editor, please visit http://www.momentumpress.net/contact

Announcing Digital Content Crafted by Librarians Momentum Press offers digital content as authoritative treatments of advanced ­engineering topics by leaders in their field. Hosted on ebrary, MP provides practitioners, researchers, faculty, and students in engineering, science, and industry with innovative electronic content in sensors and controls engineering, advanced energy engineering, manufacturing, and materials science. Momentum Press offers ­library-friendly terms: • • • • • •

perpetual access for a one-time fee no subscriptions or access fees required unlimited concurrent usage permitted downloadable PDFs provided free MARC records included free trials

The Momentum Press digital library is very affordable, with no obligation to buy in future years. For more information, please visit www.momentumpress.net/library or to set up a trial in the US, please contact [email protected].

Analytics and Tech Mining for Engineering Managers

Create your own Customized Content Bundle — the more books you buy, the higher your discount!

This book offers practical tools in Python to students of innova-

THE CONTENT

THE TERMS • Perpetual access for a one time fee • No subscriptions or access fees • Unlimited concurrent usage • Downloadable PDFs • Free MARC records

tion, as well as competitive intelligence professionals, to track new developments in science, technology, and innovation. The book will appeal to both—tech-mining and data science audiences. For tech-mining audiences, Python presents an appealing, all-in-one language for managing the tech-mining process. The book is a complement to other introductory books on the Python language, providing recipes with which a practitioner can grow a practice of mining text. For data science audiences, this book gives a succinct overview over the most useful techniques of text mining. The book also provides relevant domain knowledge from engineering management; so, an appropriate context for analysis can be created. This is the first book of a two-book series. This first book ­discusses the mining of text, while the second one describes the analysis of text. This book describes how to extract actionable intelligence from a variety of sources including scientific articles, patents, pdfs, and web pages. There is a variety of tools available within Python for mining text. In particular, we discuss the use of pandas, BeautifulSoup, and pdfminer. Scott W. Cunningham is an associate professor at the Delft University of Technology. He teaches and researches topics including data ­science, network science, and game theory. His research is directed toward helping national governments anticipate the potentially ­unforeseen consequences of new and emerging technologies. Prior to joining Delft University of Technology, he worked for AT&T as a knowledge discovery analyst, helping customers in the manufacturing and commercial sectors make the best use of their data. Jan H. Kwakkel is an assistant professor at Delft University of Technology. His research focusses on model-based support for ­decision making, with a particular focus on the treatment of u ­ ncertainty. Text mining, and more general machine learning ­techniques, are

For further information, a free trial, or to order, contact:  [email protected]

­important in his research. He has applied his research in various domains including transport, water, and energy.

ISBN: 978-1-60650-510-6

Analytics and Tech Mining for Engineering Managers

• Manufacturing Engineering • Mechanical & Chemical Engineering • Materials Science & Engineering • Civil & Environmental Engineering • Advanced Energy Technologies

Scott W. Cunningham • Jan H. Kwakkel

CUNNINGHAM • KWAKKEL

EBOOKS FOR THE ENGINEERING LIBRARY

ENGINEERING MANAGEMENT COLLECTION C.M. Chang, Editor

Analytics and Tech Mining for Engineering Managers

Scott W. Cunningham Jan H. Kwakkel