How to Measure Health Outcomes: A Hands-On Guide to Getting Started 1009240935, 9781009240932

Citation preview

How to Measure Health Outcomes

Published online by Cambridge University Press

“Supporting people to live their lives to the fullest demands understanding, defining and measuring the health outcomes that matter most to individuals and their families. The book provides an excellent guide to this important journey written by one of the most practically experienced health care professionals in the field.” Christina R Åkerman, MD, PhD, Exec. MBA Former President of the International Consortium for Health Outcomes Measurement (ICHOM)

“A must-read guide for anyone who is looking to start measuring and improving health outcomes. Written by one of the top experts in the field, this book offers practical advice that will help you overcome implementation challenges in different health care settings.” Marcia Makdisse, MD, PhD, MBA, VBHC Green Belt, MSc Academia VBHC, Brazil.

“To truly transform health care, we must understand the outcomes that matter to individuals and families and then measure those outcomes to know if we are helping. Kathleen Carberry’s How to Measure Health Outcomes provides a critical toolkit for anyone committed to delivering better health for each and for all.” Alice Andrews PhD Director of Education, Value Institute for Health and Care, The University of Texas at Austin

“This handbook is essential for achieving high value health care. Measuring the outcomes that matter to patients enables clinicians to ensure that they achieve their purpose of helping and healing. Prof. Carberry brings deep expertise and offers specific and actionable steps for how to measure what matters to the people you serve.” Elizabeth Teisberg, PhD Professor, Cullen Trust Distinguished University Chair in ValueBased Health Care, University of Texas, Austin

Published online by Cambridge University Press

“A dog-eared bookmarked version of this guide belongs in the hands of every type of clinician, health care administrator and QI enthusiast. Kathy Carberry’s decades of experience and work with the Value Institute provides an invaluable compass on the journey to creating value for patients in any health system.” Shannon Jackson, MD FRCPC MSc(HCT), Hematologist and Physician Lead for Value Based Health Care, Providence Health Care, Vancouver, Canada

Published online by Cambridge University Press

Published online by Cambridge University Press

How to Measure Health Outcomes A Hands-On Guide to Getting Started Kathleen E. Carberry University of Texas at Austin

Published online by Cambridge University Press

Shaftesbury Road, Cambridge CB2 8EA, United Kingdom One Liberty Plaza, 20th Floor, New York, NY 10006, USA 477 Williamstown Road, Port Melbourne, VIC 3207, Australia 314–321, 3rd Floor, Plot 3, Splendor Forum, Jasola District Centre, New Delhi – 110025, India 103 Penang Road, #05–06/07, Visioncrest Commercial, Singapore 238467 Cambridge University Press is part of Cambridge University Press & Assessment, a department of the University of Cambridge. We share the University’s mission to contribute to society through the pursuit of education, learning and research at the highest international levels of excellence. www.cambridge.org Information on this title: www.cambridge.org/9781009240932 DOI: 10.1017/9781009240925 © Kathleen E. Carberry 2023 This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press & Assessment. First published 2023 A catalogue record for this publication is available from the British Library. A Cataloging-in-Publication data record for this book is available from the Library of Congress. ISBN 978-1-009-24093-2 Paperback Cambridge University Press & Assessment has no responsibility for the persistence or accuracy of URLs for external or third-party internet websites referred to in this publication and does not guarantee that any content on such websites is, or will remain, accurate or appropriate. .................................................................................................................................................. Every effort has been made in preparing this book to provide accurate and up-to-date information that is in accord with accepted standards and practice at the time of publication. Although case histories are drawn from actual cases, every effort has been made to disguise the identities of the individuals involved. Nevertheless, the authors, editors, and publishers can make no warranties that the information contained herein is totally free from error, not least because clinical standards are constantly changing through research and regulation. The authors, editors, and publishers therefore disclaim all liability for direct or consequential damages resulting from the use of material contained in this book. Readers are strongly advised to pay careful attention to information provided by the manufacturer of any drugs or equipment that they plan to use.

Published online by Cambridge University Press

Contents Preface vii Acknowledgments ix

Introduction

1

1

Measuring the Outcomes of Health Care 7

2

Why Measure Outcomes? 11

3

Where to Start?

4

Identifying Outcome Measures 29

5

Collecting and Analyzing Outcome Data 36

6

Scaling Outcome Measurement 56

7

Now the Journey Begins

17

67

Index 68

vii

Published online by Cambridge University Press

Published online by Cambridge University Press

Preface This how-to guide is intended for a broad audience of people working in health care, from direct care providers (e.g., doctors, nurses, therapists) to health care administrators such as hospital executives and health insurance company employees. It is written from my perspective as someone who has helped individuals and teams in inpatient, outpatient, and health plan settings measure health outcomes. This guide focuses on practical advice for measuring outcomes in a variety of health care settings. There are other instructional resources about outcome measure development, conducting outcomes, and/ or health services research. Over the years, I have watched people who want to measure outcomes get stuck because they didn’t have basic information about how to get started – how to turn their aspiration of measuring outcomes into real information that they could use to improve their practice. My hope is that this guide becomes a regular reference for you as you embark on the process of measuring outcomes. This is a process and not a “one-and-done” undertaking. I have written this guide as though I were giving advice to an individual. The tone, therefore, is conversational. The guide draws from different conversations I’ve had over the past 20 years about the nuts and bolts of measuring outcomes. I will share with you what I’ve learned from my measurement experience throughout my career – from measuring outcomes of relatively rare conditions to measuring outcomes of more prevalent ones – including the techniques that I applied in each setting, as well as strategies to scale programs within organizations. While different medical conditions and/or care processes may require specific data points, the general steps to measuring outcomes are the same. In Chapter 6, I will describe ways that health care provider organizations can enable outcome measurement for individual clinical providers. In discussing outcome measurement, it is important to acknowledge that we are not just talking about biomedical data. Rather, this guide will teach you how to measure data in a way that tells a broader story about a person’s health that includes their own perception and experience of what health means. While this guide does not focus on broad measures of public health, when thinking about measuring health outcomes it is important to keep in mind that health is largely impacted by factors outside of the health care milieu, such as where we live and what we eat. You’ll notice throughout the guide that I generally refer to patients as people. While the word “patient” reminds us of whom we serve, there are also connotations from which we should move away. One of the original meanings of patient was “one who suffers,” or “enduring without complaint,” ix

https://doi.org/10.1017/9781009240925.001 Published online by Cambridge University Press

x

Preface

based on the word’s Latin root, patiens, which implies a sense of passiveness.1 Additionally, the word patient can imply a hierarchical and patriarchal social system that immediately puts the care provider a step above the patient as “more knowing.” In fact, both bring knowledge that is needed to improve the patient’s health. While the care provider knows more about certain areas, such as the disease process and biology, patients know much more about how the condition affects their daily lives, what it keeps them from doing, and so on. In the same way, using the word “member” to describe someone who is part of an insurance plan obscures a person’s individual agency and contributes to a one-size-fits-all mentality. It can imply someone who incurs costs that the insurer or employer has to pay or not pay for. The word distances us from the person who needs the services to better their health and live the life that they want to live. So, in this guide, I will refer to patients and members simply as “people” because, in fact, that is what they are: people like you and me. People whom we have the honor of caring for each and every day. People who need our help. We are also going to consider the voices of family members and caregivers because measuring their outcomes is also critical. Poor caregiver outcomes can directly impact both patient outcomes and caregiving in and of itself by putting people at risk for higher levels of stress, depression, and anxiety.2,3 And one final note, I am writing this from my experience working in the US health care system. My hope is that the nature of this how-to guide will lend itself to being adapted and applied within the health care system where you live and work, and I send you my very best wishes.

References 1. J. Neuberger. Let’s Do Away with “Patients.” BMJ 1999; 318: 1756. 2. J. T. Bidwell, K. S. Lyons, C. S. Lee. Caregiver Well-being and Patient Outcomes in Heart Failure: A Meta-analysis. J Cardiovasc Nurs 2017; 32: 372–382. 3. M. Bevans, E. M. Sternberg. Caregiving Burden, Stress, and Health Effects among Family Caregivers of Adult Cancer Patients. JAMA 2012; 307: 398–403.

https://doi.org/10.1017/9781009240925.001 Published online by Cambridge University Press

Acknowledgments

How to Measure Health Outcomes: A Hands-On Guide to Getting Started is a reality thanks to many people who have helped me in countless ways. I was inspired to write it after reading a book about writing that my colleague, Victoria Davis, lent me. As I talked about the idea of writing this guide with my other colleagues, Amy Madore, Sarah O’Hara, and Kasey Ford, their enthusiasm and encouragement gave me the momentum to write a draft for the Measuring Outcomes That Matter course in the first year of the Master of Science in Health Care Transformation degree program. It was an ambitious goal. With their help and the support of Alice Andrews, the director of education, and Elizabeth Teisberg and Scott Wallace, the directors of the Value Institute for Health and Care, I am pleased to share this guide with you. I cannot thank Sarah O’Hara enough for being my thought partner on how best to organize this work and for asking the right questions that got me thinking more about what I needed to say, Elizabeth Teisberg for her thoughtful review of the manuscript, nor Amy Madore for her superb editing skills and ability to ask questions that stretch my thinking. Vital to measuring outcomes is understanding what the outcomes are that matter most to people. Hence, I am thankful for the critically important work of Elizabeth Teisberg and Scott Wallace in developing the Capability, Comfort, and Calm outcome measurement framework. This framework anchors outcome measurement in the outcomes that matter most and refocuses the health care system on achieving better outcomes for the people it serves. From my clinical vantage point, this simple and elegant framework fills a void in the world of outcome measurement in health care. I am deeply grateful for their thought leadership, colleagueship, and everything I have learned from them. I am also so grateful to my colleague, Chris Ulack, who keeps me laughing; Joel Suarez, who taught me that writing is thinking; and the entire Value Institute team for being such encouragers and cheerleaders for these outcomes work. There are several organizations mentioned in this guide that are exemplars of outcome measurement. My teammates and I at the Value Institute are grateful to them for sharing their stories and experiences with us so that they can be disseminated more broadly to drive much needed improvement in health care.

xi

Published online by Cambridge University Press

xii

Acknowledgments

And finally, I have to thank my family: my husband, Michael, for being my best friend and for all of the extra carpool he did and dinners he made so that I could focus on writing; my parents and sister for always believing in me; my in-laws for their encouragement; and my children, Brigitte and Daniel – thank you for your unconditional love and patience, even when there were nights I had to skip stories. I love you so much. You inspire me every day.

Published online by Cambridge University Press

Introduction

The purpose of health care is to improve people’s health in a caring manner. A challenge to delivering on this purpose is that health care professionals often don’t know if the care that they deliver or pay for improves people’s health. And it’s not for lack of trying to measure – health care is awash with measures. In fact, many health care providers feel like they are drowning in measures and reporting requirements, and that is because they are! Your organization probably tracks a number of indicators that are required by regulators or payers, or because they are linked to internal quality-improvement initiatives. But take a step back and ask yourself how many of those measures actually measure the impact on patient health during or after care? Odds are, the answer is “not many.” Instead, most existing measures focus on assessing how individual health care providers, clinics, and hospitals are performing. The measures don’t focus on answering the question of how people’s health has or has not improved, although this is starting to change as the use of patient-reported outcome measures grows.1 And simply complying with reporting requirements is not enough. The idea of measuring the results of health care and focusing on patient outcomes is not a new one. Florence Nightingale and Ernest Codman demonstrated the importance of this in the mid-to-late 1800s2,3 – yes, the 1800s! Unfortunately, the idea did not gain traction, and here we find ourselves in the twenty-first century still struggling to make patient outcome measurement commonplace. To get us back on the path to measuring the results of health care – the patient’s outcomes – it is important to understand how we have lost our way. This starts with a brief visit to the history and evolution of measurement in health care. Once we have done that, we can reorient ourselves to the measurement of outcomes and pursue it fervently!

The Current Health Care Measurement Landscape The current measurement landscape is characterized by countless measures that are designed in order to evaluate the “quality” of health care. What does “quality” in health care really mean? 1

https://doi.org/10.1017/9781009240925.002 Published online by Cambridge University Press

2

How to Measure Health Outcomes

In 1966, Avedis Donabedian conceptualized a framework for assessing the quality of medical care that came to be known as the Donabedian framework.4 It is organized into three categories: structure, process, and outcome. Donabedian described structural measures as measures to evaluate the setting in which care is delivered that may include such measures as the number of board-certified physicians in a clinic or hospital, adequacy of the facilities, and so on. Process measures measure the processes of care, such as the time it takes to get from the emergency room to the cardiac catheterization lab for patients presenting with an acute myocardial infarction (heart attack). Outcomes include the measurement of survival, the degree of recovery from illness, and/or the regaining of function in an area where it was lost.5 While Donabedian described outcomes in a way that aligns with how we think about patient-centered outcomes today, current measurement efforts reflect a focus on structure and process measures, and usually define outcomes as measures of clinicians’ outcomes – that is, how well they have performed in delivering care. This may be the result of a narrow interpretation of Donabedian’s framing around medical quality assessment as the need to evaluate providers of medical care instead of the receivers of medical care. Defining quality in health care as achieving success in all three categories – structure, process, and outcome – makes measuring it and differentiating it from outcomes challenging. What needs to be made clear in our thinking about quality is that the ultimate measure of quality in health care is improved outcomes. The structure and process measures in Donabedian’s framework are the inputs of the health care system. Conversely, measures in the outcomes category are the outputs of health care. Measuring and improving the inputs of care delivery should only be done to the extent that it improves the output: outcomes for individuals receiving care. Typically, inputs are easier to collect as they are more clearly time-bound and straightforward and don’t necessarily require information from patients. Measuring processes, in particular, provides a way to account for the application of the best medical knowledge to-date, knowing that the ideal outcomes may not be attainable because of the limitations of medical science at any given point in time. To measure the true and longitudinal outcomes of health care delivery at the time that Donabedian published his framework seemed daunting. With the vast array of technology available today to support longitudinal outcome measurement, and with the increasing acceptance of patientcentered outcome measurement, feasibility should no longer be considered a major barrier to measuring outcomes. Nevertheless, longitudinal outcome measurement remains rare in practice. Today we find ourselves swimming in a sea of certifications and process measures, without at least a commensurate – and ideally unyielding – focus on

https://doi.org/10.1017/9781009240925.002 Published online by Cambridge University Press

Introduction

3

measuring outcomes.6 Part of this proliferation is not without reasonable cause and positive intent. Certifications and processes are important inputs of the health care system that should be optimized. But they do not guarantee and may not even correlate with good health outcomes. In an effort to increase safety and promote quality in health care, the US federal government started requiring quality measurement in the late 1990s, when the field of quality improvement was still very nascent. In 1998, President Bill Clinton established the Advisory Commission on Consumer Protection and Quality in the Health Care Industry. This commission’s charge was to lead a national effort to improve health care quality in the USA. It led to the creation in 1999 of the National Quality Forum (NQF), a national clearinghouse for consensus-based, standardized quality measures. In addition to setting national quality standards, the NQF convened a Strategic Framework Board in order to develop a strategy for a national quality reporting system and identify priority areas and potential barriers.7 The NQF currently oversees a plethora of measures, although not enough of them are patient-centered outcomes.

Reorienting to the Measurement of Patient Health In an effort to reorient ourselves and create a foundation for measuring the outcomes of health care, let’s start by defining some key terms: health, health care, and outcomes. What is health? According to the World Health Organization, health is “a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity.”8 Health care, then, constitutes the efforts to maintain or restore health, usually by trained and licensed professionals. And an outcome is the result or consequence of an action – or the absence of action. So, if the purpose of health care is to maintain or restore health, we need to measure in a way that tells us if the care that we provide achieves that goal. Simply put, does care help improve health? Scott Wallace and Elizabeth Teisberg developed the Capability, Comfort, and Calm outcome measurement framework that enables those who work in health care to measure results on these three important dimensions for people receiving health care.9 We will use this framework throughout the how-to guide and look at it in depth in Chapter 1.

Why Are Patient Outcomes Not Measured More? Frankly, it is hard to get started. The questions of which measures to use and how to collect the information are stymying for many. In this guide, we will tackle these and the many other questions that arise. Building an outcome measurement program may seem overwhelming, but it need not be. This guide will walk you through where to begin and how to overcome the most common challenges that you will face.

https://doi.org/10.1017/9781009240925.002 Published online by Cambridge University Press

4

How to Measure Health Outcomes

For a variety of reasons, health care providers as a whole have not placed a high priority on outcome measurement. Although clinicians usually ask informally how a person is feeling, that conversation is rarely part of the patient’s medical record or measurement approach. Instead, providers often have to track measurements required by regulators or payers, such as readmissions and health screenings, but rarely push the conversation further. Or they measure the results of process improvement initiatives (e.g., reducing door-toballoon time for heart attack patients), but not the extent to which those initiatives actually led to better health (e.g., decreased mortality, days at work, improved quality of life). In some cases, clinicians may not measure outcomes because they think that they already know how patients are doing and do not realize how much more they might learn from an in-depth assessment of care results. This indepth assessment can increase the aspirations of a clinician or care team by giving them insight into how their care impacts outcomes and how they can improve. This cycle of measurement and improvement creates a culture of learning and reinforces the focus of the team on improving outcomes.

Your Personal Reflections Before We Get Started People work in a variety of capacities within the health care system. As you embark on measuring outcomes, it is important to reflect on why you want to start measuring outcomes and what impact you hope to make by doing so. This how-to guide will provide guidance in different ways depending on the role that you play in health care. • If you are an individual clinical team member, it will provide you with steps to measure the health of your patients in order to determine if they are getting better and/or staying well physically, mentally, and emotionally, and to assess how the process of accessing and receiving care affects their daily lives. • If you are a leader of a health care delivery organization, you will learn measurement steps and techniques to determine what kind of results your organization is achieving in terms of improving the health of the people seeking care from you. • If you work in a health plan, you can also apply these measurement techniques to answer two fundamental questions: Are the people we cover improving their health? How are the provider organizations we contract with helping people improve health? • If you are a researcher, this guide will offer ways for you to think about what outcome you are trying to influence through your work. Throughout my career, I have reviewed many clinical research proposals designed to answer research questions that are very interesting but will have little

https://doi.org/10.1017/9781009240925.002 Published online by Cambridge University Press

Introduction

5

impact on people’s health – or that don’t even consider the potential impact on outcomes. If you don’t identify with any of these roles in health care, take a few minutes to think of a health care experience that you or a loved one has had. What was the reason for interacting with the health care system? What did you or your loved one need? What questions did you or they have, and were they answered? And how did your experience and the outcomes line up with what you needed or hoped for? Regardless of which role you identify with, take a few moments to reflect on why you want to measure outcomes. Jot down your reasons in the space below.

Before we dive into getting started with outcomes measurement, you need to ask yourself another important question. Where do I want to go with outcome measurement? What is my goal? As Stephen Covey says, “Begin with the end in mind.”10 The path to measuring outcomes can be wrought with distraction, especially in the age of big data and sexy data collection and analytic tools that don’t necessarily deliver on making outcome measurement easier. If you don’t start with a clear objective or purpose, it will be hard to stay on the path. Covey suggests creating a mental model of what you want to achieve, then setting out to bring that mental model to life. This will be critical to your success and help you stay focused on reaching your objectives. When I led an outcomes team for children’s heart surgery, our end goal was to have valid, reliable, and reproducible outcome data that the surgeons could use during surgical consultation. We also wanted to share those data publicly to help people know what they could expect when they came to us for surgery. Once we had set these shared goals, the team and I looked to a leader in health outcome transparency – The Cleveland Clinic – for examples of how to operationalize the work ahead. We kept copies of and referred often to their outcomes books, which provided a strong visual representation of our end goal. We also used them as an example any time that we needed to make the case to others for the imperative, feasibility, and promise of investing in outcome measurement. In this way, The Cleveland Clinic’s work helped bring us one step closer to a shared mental model of what is possible.

https://doi.org/10.1017/9781009240925.002 Published online by Cambridge University Press

6

How to Measure Health Outcomes

Spend a few minutes thinking about what your end goal is for outcome measurement. What do you see yourself doing with the data you collect? Close your eyes and try to picture it. Now write down the end goals that came to mind. You can also draw a picture if that is more helpful to you. Stick figures are welcome!

Ready to start? Here we go!

References 1. P. R. Deshpande, S. Rajan, B. L. Sudeepthi, C. P. Abdul Nazir. Patient-Reported Outcomes: A New Era in Clinical Research. Perspect Clin Res 2011; 2: 137–144. 2. D. Neuhauser. Florence Nightingale Gets No Respect: As a Statistician That Is. BMJ Qual Safe 2003; 12: 317. 3. J. H. Talbot. Obituary, Ernest Amory Codman (1869–1940). N Engl J Med 1941; 224: 296–299. 4. A. Donabedian. Evaluating the Quality of Medical Care. Milbank Q 2005; 83: 691–729. 5. M. Best, D. Neuhauser. Avedis Donabedian: Father of Quality Assurance and Poet. BMJ Qual Safe 2004; 13: 472–473. 6. E. Teisberg, S. Wallace. The Quality Tower of Babel. Health Affairs Blog. www .healthaffairs.org/do/10.1377/forefront.20150413.046869. Published April 13, 2015. Accessed August 16, 2022. 7. H. Burstin, S. Leatherman, D. Goldmann. The Evolution of Healthcare Quality Measurement in the United States. J Intern Med 2016; 279: 154–159. 8. Constitution. World Health Organization. www.who.int/about/governance/constitu tion. Accessed August 16, 2022. 9. S. Wallace, E. Teisberg. Measuring What Matters: Connecting Excellence, Professionalism, and Empathy. Brain Injur Profess 2016; 12: 12–15. 10. S. R. Covey. The 7 Habits of Highly Effective People: Powerful Lessons in Personal Change. Fireside Press; 2004.

https://doi.org/10.1017/9781009240925.002 Published online by Cambridge University Press

Chapter

1

Measuring the Outcomes of Health Care

The outcomes of health care can be defined in many ways. In the biomedical sense, outcomes represent the result of a disease process. Biomedical research focuses on understanding the mechanisms behind diseases so that interventions can be made to improve outcomes by changing the natural course of the disease. Outcomes can also refer to the end results of care, which you measure to assess the effectiveness of a particular clinical management approach, either medical or surgical. These outcomes can be positive or negative. Our understanding of what constitutes a “good” or “successful” outcome can change over time as medical care improves. For example, for children born with critical heart disease, the natural history of the disease has been death in infancy. Since the first surgery was performed on a child with critical heart disease in 1944, innovations in surgical treatment strategies for congenital heart disease have altered children’s survival trajectory significantly. Today, more children survive than die from congenital heart disease. Survival has become the expectation and, as a result, attention has shifted to children’s developmental and physical abilities throughout their lives. This is true of cancer as well, as now many cancers are viewed as chronic diseases. So, with medical advances comes an evolution of what success means for a person needing help with a health problem. In addition to medical advances, an evolution in our expectations for wellbeing and quality of life has also influenced what success looks like in health care. A good example is when hospice care is viewed as an option for people who may not want extreme end-of-life efforts using treatments with unknown results. Sometimes the outcomes are unknown for different treatment modalities or there are choices among treatments, so patient preferences and values come into play. It’s important to recognize that as medical science and health care evolve, so will the outcomes that we measure.

Measuring the Outcomes That Matter Most to People Elizabeth Teisberg and Scott Wallace developed the Capability, Comfort, and Calm outcome measurement framework after aggregating a decade’s worth of research from talking to people about their lived experience 7

https://doi.org/10.1017/9781009240925.003 Published online by Cambridge University Press

8

How to Measure Health Outcomes

managing their health.1 A part of the discussion included learning about the outcomes that mattered most to them. What they found was that the outcomes that matter most to people tend to fall into three domains: capability, comfort, and calm. Capability is your ability to do the things in life that are important to you – the things that make you you. Better functional outcomes enable you to live the life that you want to live. Here are some examples of capability: 1. Running a marathon after an ACL (anterior cruciate ligament) tear 2. Walking your child to school without feeling dizzy 3. Being intimate with a partner 4. Being able to see in order to read and drive 5. Going to work 6. Dancing at your child’s wedding Comfort is freedom, to the extent possible, from pain or emotional and mental suffering. If we look again at the World Health Organization’s definition of health,2 we can see how achieving capability and comfort align to physical, mental, and social well-being respectively. Calm is the ability to receive care in the least disruptive way possible during the course of daily life. In other words, it’s less mayhem, less wasted time, and less hassle. It reflects patient experience and extends beyond the hospitality of the clinic or hospital to the experience of navigating life with the condition and care. This framework is positively framed and reorients us to improving health. It centers our efforts on what the person needs and what their goals are for health – not necessarily the disease, and not necessarily what the health care system needs. This framework enables us to orient our measurement and improvement efforts around achieving health and the outcomes that matter most to patients. It also allows us to reframe existing measurement efforts into a framework that facilitates measuring the results of health care. Furthermore, the Capability, Comfort, and Calm outcome measurement framework can be adapted through time as medicine and health care evolve. To begin measuring outcomes, you need to know which outcomes are important to people. Why did they seek help to begin with? Often one can elicit the outcomes that matter by asking people what are their health goals. Why did they come to see you? What questions are they asking? This can be applied to a variety of settings, including those in which the people seeking care are well. In the case of health screenings, for example, providers use a set of screening questions to ask children with asthma if their asthma is under control. The goal, of course, is for the child to be able to function in school and play without symptoms of asthma. The measurement here should be tracking the asthma screening score for each child to see how their asthma is being managed over time.

https://doi.org/10.1017/9781009240925.003 Published online by Cambridge University Press

Measuring the Outcomes of Health Care

9

Measure at the Individual Patient Level To determine if care helped a person, outcome measurement must be at the level where the care was provided: the individual. I can’t emphasize this enough. Many quality and patient experience measures are gathered at the level of a care unit, clinic, or doctor. For example, patient satisfaction surveys seek to measure the patient’s experience with the health care system and not the individual care delivery outcomes. While the survey items address many elements that could resemble measures of calm, they do not assess individuals with distinct medical conditions and rarely focus on life beyond the care received at a facility or through telemedicine. Instead, the survey asks individuals to assess the hospitality and other characteristics of the setting in which they received care as opposed to if and how care supported and enabled them to live their life. The US Centers for Medicare and Medicaid Services (CMS) star ratings, which gauge the performance of American hospitals, are another example of measures that do not measure at the right level or assess the results of care for a particular disease.3 This is problematic in many ways. For example, if a hospital is great at cancer care and below average at cardiac care, is it a great hospital, a below-average hospital, or an average hospital? How should a family seeking cardiac care for a loved one compare hospitals using the star rating? Health care outcomes are achieved one patient or family at a time, and this requires measurement at the individual patient level.

Measure Outcomes across the Care Cycle Keep in mind the full cycle of care when measuring. Think about what is happening in a person’s life beyond the walls of the health care setting. Often the measurement of outcomes takes place after a health care intervention to determine how things turned out. Outcomes can and should also be measured during the course of care. In speech therapy, physical therapy, and treatments for many anxiety disorders, measurements can be made to track progress during care to ensure that progress is being made toward a goal, with treatment adjusted in real time as needed. For example, a group of psychologists at Cincinnati Children’s Hospital Medical Center found a way to track session by session if children with obsessive-compulsive disorder were making progress in treatment. Children or their parents (if the children were unable) indicated how successful they were feeling at controlling or managing their condition by characterizing their ability to focus on school and work, and to enjoy playtime with friends. The care team measured these outcomes throughout the child’s care cycle to inform treatment decisions, with the goal of always aligning therapeutic interventions with patient success. And by measuring outcomes more frequently, the psychology team identified provider variation

https://doi.org/10.1017/9781009240925.003 Published online by Cambridge University Press

10

How to Measure Health Outcomes

in results, adopted best practices for achieving better results, and ultimately improved the children’s life at school, home, and play more quickly and more effectively than they had previously (A. Madore, K. Carberry. Developing a New Tool to Measure OCD Treatment Progress at Cincinnati Children’s Hospital Medical Center. Unpublished case, Value Institute for Health and Care at the University of Texas at Austin, 2020).

References 1. E. Teisberg, S. Wallace, S. O’Hara. Defining and Implementing Value-Based Health Care: A Strategic Framework. Acad Med 2020; 95: 682–685. 2. Constitution. World Health Organization. www.who.int/about/governance/constitu tion. Accessed August 16, 2022. 3. S. Wallace, E. Teisberg. Commentary: Lose Star Ratings, Instead Measure What Matters to Patients. Modern Healthcare. www.modernhealthcare.com/article/20190 112/NEWS/190109916/commentary-lose-star-ratings-instead-measure-whatmatters-to-patients. Published January 12, 2019. Accessed August 16, 2022.

https://doi.org/10.1017/9781009240925.003 Published online by Cambridge University Press

Chapter

2

Why Measure Outcomes?

Earlier in this guide, I touched on some of the reasons why you should measure individual health outcomes. I’m not going to spend a lot of time making the case for measuring outcomes, as that is not the purpose of this guide. If you have picked up this guide, you hopefully have your own answer as to why you want to measure. Still, it can sometimes be helpful to have a holistic understanding of why outcome measurement is important, particularly if you have to convince others in your organization. So, let us look at the core reasons to measure outcomes in a little more depth. There are four main reasons to measure outcomes. First, and most simply, measuring the change in outcomes tells you how you are doing with respect to providing health care. Without this information, it is nearly impossible to know whether the care you deliver is benefitting patients or not. Second, once you understand these results, you can identify opportunities for learning and improvement. Third, outcome data give patients and their families critical information about what to expect when they seek care from you or your organization (or, if you work for a payer organization or employer, from the health care providers within your network). We’ll look closer at each of these three reasons in the sections that follow, but before we do so, we need to consider a fourth foundational reason to measure outcomes. Many studies have documented the existence of unacceptable rates of preventable patient harm, regional variations in care standards and outcomes, and disparities in health results based on race, ethnicity, and other social identity characteristics. In light of these findings, you have an ethical obligation to understand whether the care that you provide is helping or harming.1–5 It should go without saying that the stakes are high in health care – people’s lives and well-being are on the line. Whether medical or surgical, treatments can be risky, and the decision not to treat or intervene also comes with both risks and benefits. We owe it to patients to know and be able to explain the likely outcomes of treatment decisions and to adjust our practices when the care we provide yields less-than-optimal results. With that said, let’s dive deeper into why measuring outcomes is so important. 11

https://doi.org/10.1017/9781009240925.004 Published online by Cambridge University Press

12

How to Measure Health Outcomes

Understanding Your Current Performance Measuring the current results of patient care is valuable regardless of where you sit within the health care system. These results – the change in outcomes for patients – reveals whether and how much the care helped. As a result, you need to measure over a period of time: before care occurs, while care is taking place, and following care. If you’re a clinician, outcome data provide baseline information about the improvements patients achieve from the care that you provide. How have your patients gained with respect to Capability, Comfort, and Calm? Did care achieve survival, restore function, or optimize wellness? Are your patients experiencing less pain, anxiety, or emotional distress than before? Do your patients feel that it is easy and seamless to get the care that they need from you? This type of information can then drive learning, improvement, and innovation because you can’t help but ask questions based on the data you collect. By having baseline outcome information, you also have a way of knowing how any new treatment modalities you try compare with results of your current practice. Some of you may have similar experience in the setting of clinical trials, where the results of a particular intervention are compared to baseline data to see if the intervention resulted in any improvement. Often, any adverse effects or complications from the study are tracked carefully during the course of the clinical trial, but once the study is completed, such careful monitoring ends. Instead of ending when a study ends, outcome tracking must become a routine part of clinical care so that care providers know – from basic preventive interventions to complex surgical procedures – if their management strategies are effective in helping patients achieve better health. If you are a hospital administrator, you know that the business of health care continues to move toward alternative payment models with a continued emphasis on quality, patient satisfaction, and improved health outcomes, all in the context of tighter margins. Health care organizations that focus on measuring outcomes will be better positioned for future payment model changes that rely on having solid outcome data. As such, it will be best if you have true outcome data to inform these payment models instead of having to rely mostly on process measures. On a micro level, from an operational standpoint, outcome data provide information to guide managerial decision-making and offer insights into which system inputs might be missing and/or not working well to achieve effective and efficient patient care. These insights can inform decisions about resource allocation and new investments. Examined at the macro level, outcome data also allow the executives and board members responsible for hospital or health center governance to meet their ethical and fiduciary duty to ensure that health care services improve the health of people in the

https://doi.org/10.1017/9781009240925.004 Published online by Cambridge University Press

Why Measure Outcomes?

13

community that they serve. Similarly, for both clinicians and health care administrators, outcome measurement highlights care disparities – areas where certain groups of people are not well-served by existing care models – that might otherwise remain unseen. It reveals unwarranted variation in care that may contribute to these disparities or undermine consistency of good results. If you work for a payer or benefit plan, outcome data can provide insight into the health of people in your plan. Are the services that you pay for helping people get the best outcomes possible? Unlike most clinicians and hospital or clinic administrators, payers have a longitudinal view of how their members interface with health care providers. Based on claims data, they have visibility into resource utilization over time that can inform questions about the efficiency and outcomes of care delivery. Employers, too, are payers and can play a significant role in asking whether the care they pay for is truly meeting their employees’ needs. Recently, for example, Walmart and other large employers have started routing employees who need care for certain conditions to a select number of regional “centers of excellence” based on data that shows that those health care organizations generate demonstrably better outcomes for lower costs.6 Knowing what your people need and if you are effectively meeting those needs is crucial in informing strategy and allocating resources. Thus, measuring outcomes supports the professionalism of all who work in health care, enabling alignment of effort and resources with the purpose of helping people.

Identify Opportunities for Learning and Improvement The second reason to measure outcomes is that once you have a foundational understanding of how you are doing, you can see where you can do better. This leads to learning and innovation. Take, for example, the Martini Klinik, a world-renowned center for prostate cancer care in Hamburg, Germany. Clinic providers measure key patient-centered outcomes of prostate cancer surgery – including continence and potency – for every patient that they see. They have a robust database that was started in the late 1990s and that now includes functional outcomes data, genomic data, and other clinical and administrative data for the thousands of patients that they have treated over the years. From these data, they know at any point in time how their surgeons and teams are performing and can readily identify opportunities to improve care. The opportunities may present themselves in clinical care (e.g., a surgical technique that needs to be refined); in the decision to pursue new research because the knowledge needed to address an identified problem doesn’t yet exist; or in other changes to the care delivery model.7 When teams measure together and improve, a sense of pride emerges. A private-practice electrophysiologist in Alabama who has built an extensive

https://doi.org/10.1017/9781009240925.004 Published online by Cambridge University Press

14

How to Measure Health Outcomes

outcome measurement program describes how, initially, the office staff and clinical team were somewhat skeptical about investing the time to capture outcome data. Once they started seeing the results of their care and how it could improve, they felt energized. As a result, the atmosphere of the whole practice changed, and patients remarked on how “different” (in a good way) their office felt (J. Osorio, Your Outcome Measurement Journey [personal interview, December 17, 2019], Austin, TX).

Providing Critical Information to Patients and Families Speaking of patients, the third reason to measure outcomes is that doing so provides patients and families with vital information about what to expect when they seek care and enables informed conversations between patients and clinicians about treatment options. Patients want to know what a care provider’s personal experience is with treating a particular condition. They may not say this explicitly, but they are trusting that when care providers quote statistics about results, those results reflect their own. For example, the Alabama electrophysiologist that I mentioned said that he wasn’t comfortable using clinical trials data to counsel patients about treatment modalities if he couldn’t also provide the results that he had achieved with his own patients. Consider the example of a primary care provider or cardiologist who prescribes a statin for a patient with high cholesterol who is not responsive to diet and exercise changes. You can imagine that the clinician would discuss the risks and benefits of taking statins with the patient. What if the patient were to ask: “How many of your patients take this medication, and how are they doing on it?” How many clinicians are prepared to answer this question? In addition to knowing the most up-to-date general information about statins, the prescribing clinician could track each of his or her own patients on a statin and be able to say something like: “I’ve prescribed this medication to x patients over the last year, and less than x percent of them experienced x.” Often statistics quoted during clinical consultations represent data from research studies and not direct outcomes of the care provider in question. Those studies show results under specified conditions that may not hold for an individual patient. Moreover, the study protocol is often not the clinical protocol that most care providers follow outside of the study. So, not only are the quoted statistics not the results of patients seen by a particular clinician, but they may also not reflect the results of the clinical protocols used by that particular care team. While study data can certainly inform the provider’s decision to recommend a treatment, it can be misleading to cite those statistics when discussing the treatment with patients, even if that is not the intent.

https://doi.org/10.1017/9781009240925.004 Published online by Cambridge University Press

Why Measure Outcomes?

15

An extreme example of this misrepresentation occurred some years ago when in vitro fertilization (IVF) clinics began to open more widely across the USA, and many advertised generalized study results to attract patients. In some cases, clinics that had never succeeded in enabling anyone to become pregnant provided prospective patients the average success rates from national studies. It became a scandal, and Congress intervened. Such advertising has since become illegal, and IVF clinics now must report their own outcomes publicly.8 While most other examples are less egregious, the data on IVF make the point clear. When it comes to decision-making about their health, patients and families should have accurate information – including scientific evidence for similar patients as well as the specific results from patients cared for by a particular physician or team. Of course, making the decision to measure outcomes is a significant one. At this point in our current health care environment, there are very few external incentives for measuring the outcomes that matter most to patients. While there have been some attempts to create these incentives (e.g., the employer-sponsored centers of excellence programs described earlier), these incentives are not yet far-reaching, and are probably not the most effective way to motivate people to measure outcomes. Making the decision to measure outcomes really comes from internalizing the importance of having outcome data for the reasons listed, and thereby making a commitment to measure. For many clinicians, this idea attaches to their calling and professionalism. This internal commitment is also important because once you measure, you may not like what you find. Outcome data may require you to give up long-held beliefs about care or to allocate scarce resources to investments needed to improve poor results. Taking to heart an understanding of why measurement matters can help you move forward despite these challenges.

References 1. D. H. Havens, L. Boroughs. “To Err Is Human”: A Report from the Institute of Medicine. J Pediatr Health Care 2000; 14: 77–80. 2. The Dartmouth Atlas of Health Care. www.dartmouthatlas.org/. Accessed August 16, 2022. 3. E. O. Teisberg, M. E. Porter. Redefining Health Care: Creating Value-Based Competition on Results. Harvard Business Review Press, Boston, MA, 2006. 4. M. A. Winker. Measuring Race and Ethnicity: Why and How? JAMA 2004; 292: 1612–1614. 5. D. Thompson. Framing the Dialogue on Race and Ethnicity to Advance Health Equity: Proceedings of a Workshop. The National Academies of Sciences, Engineering, and Medicine, Washington, DC, 2016.

https://doi.org/10.1017/9781009240925.004 Published online by Cambridge University Press

16

How to Measure Health Outcomes

6. M. Gamble. The 17 Health Systems to Which Walmart Sends Employees for Care in 2021. Becker’s Hospital Review. www.beckershospitalreview.com/strategy/the-17health-systems-to-which-walmart-sends-employees-for-care-in-2021.html. Updated June 11, 2021. Accessed August 16, 2022. 7. M. E. Porter, J. Deerberg-Wittram, T. W. Feeley. Martini Klinik: Prostate Cancer Care 2019. Harvard Business School Publishing, Boston, MA, 2019. 8. M. E. Porter, S. Rahim, B. C.-S. Tsai. In-Vitro Fertilization: Outcomes Measurement. Harvard Business School Publishing, Boston, MA, 2008.

https://doi.org/10.1017/9781009240925.004 Published online by Cambridge University Press

Chapter

3

Where to Start?

Once people have decided to measure patient outcomes, their next question is typically, “Where do I start?” Answering that question can feel very daunting, but it does not need to be. First, let’s establish some basic vocabulary for outcome measurement and revisit the definition of an outcome. Outcomes are the result or consequence of an action or absence of action. The outcomes of health care are the results of health care delivery services or the absence of those services and their effects on health. So, outcomes are the change that occurs, in this case, for the patient or family. As described in Chapter 1, the outcomes that matter most to people fall into three categories: Capability, Comfort, and Calm. Once you identify the outcomes that matter most (discussed at the end of this chapter), you need to measure to what extent those outcomes were achieved or not. This involves identifying existing measures or defining your own. Outcome measures are standard units of analysis used to express the degree to which the outcome was achieved. I will go into more detail on how to identify outcome measures in Chapter 4. The first step to measuring outcomes is to identify the cohort or segment of people whose outcomes you want to understand. Once you have done this, there are two options for next steps. First, if you have some idea of the outcome measures you might use and have data readily accessible, you might find it helpful to gather some baseline data on how the cohort or segment you identified is doing now. It is important to note that you might not have baseline data for everything you’d like to measure. Don’t let that stop you from getting started; it is not necessary at this point. If you have no idea which measures to use, the best next step after identifying your cohort or segment is to determine the outcomes that matter most to those people. Even if you do start with gathering baseline data, you will still want to identify the outcomes that matter most to the people you serve. You will likely embark on that step very soon after analyzing your baseline data. These steps apply whether you are a clinical care provider, health care administrator, or someone who works on the payment side of health care. Let’s look at each step in detail. 17

https://doi.org/10.1017/9781009240925.005 Published online by Cambridge University Press

18

How to Measure Health Outcomes

Step 1: Identifying the Cohort or Segment A cohort is a group of people who share a common characteristic or experience. In health care, cohorts are often defined by medical conditions. Cohorts can be further refined into smaller groups, or segments, based on identified shared needs. For example, children with congenital heart disease represent a cohort of patients. Within that cohort, children with anomalous aortic origin of the coronary artery (AAOCA) constitute a segment because while these children have congenital heart disease, their needs are meaningfully different from those children with other types of congenital heart disease (Figure 3.1). For example, I spoke with a mother of a child with AAOCA who sought support from a local congenital heart disease support group. Most of the parents in the group had children who were diagnosed either at birth or prenatally and had thus had several years to process the diagnosis. This mother had just learned of the diagnosis in her preteen child, whom she had always thought was healthy, and was struggling to come to terms with this new reality. In her words, she couldn’t “relate” to the other parents because they were at different points in their care journey and had different needs. Typically, if you have some idea about the unmet needs of people you serve, you can readily identify a patient segment or segments. Otherwise, you are likely to start more broadly with a cohort of patients. When the congenital heart surgery team I worked with started its outcome measurement journey, we wanted to understand the outcomes of all patients who had surgery. Over time, more in-depth outcomes tracking for the AAOCA segment was initiated alongside the development of a focused clinical program designed to meet the unmet needs of patients with AAOCA and their families.

Children with congenital heart disease

Figure 3.1 Venn diagram of children with congenital heart disease (cohort) and children with AAOCA as a segment within that cohort.

Children with AAOCA

https://doi.org/10.1017/9781009240925.005 Published online by Cambridge University Press

Where to Start?

19

For Clinicians Think about the patients for whom you provide care and the different medical conditions you treat. Perhaps you want to measure and improve outcomes for a medical condition that is common in your practice, or perhaps you want to create a distinctive clinical program that addresses the needs of a small but still important patient segment. Either way, you need to explicitly define for which patient cohort or segment you want to start tracking outcomes. Sometimes getting started is as simple as making sure you can answer a few basic questions that your patients may already be asking you. For example, in congenital heart surgery, some of the questions that often came up in surgical consultation were: 1. How many patients with this problem have you taken care of? 2. How good are you at taking care of this problem? 3. How do you compare to others who also do this surgery? 4. How much will this cost? Reflect on your practice. Who are you taking care of? What questions are they asking you, and can you honestly answer them with data from your own experience? Is there an area where you suspect that you can deliver more stateof-the-art care that achieve better results?

For Health Care Administrators Those in charge of an entire practice, service line, or other health care organization can use a process like the one for clinicians to identify a cohort or segment for measuring outcomes; they just need to do it on a broader scale. Think about the people and communities that you serve. What specific health care services and programs do you offer and for whom? The “for whom” is your cohort. For example, a primary care clinic may see people with diabetes, high-blood pressure, asthma, and so on. People with these medical conditions are your cohorts. Within those cohorts, you can then identify specific segments as appropriate. As a health care administrator, you have a key role in enabling the process of measuring outcomes and collaborating with clinicians to examine the questions posed in the “For Clinicians” section. Another way to support these efforts is to make outcome measurement an organizational priority and allocate human and technical resources to it. Just like clinicians, you are accountable to the people that you serve to know what your outcomes are. For those clinicians in your practice who aren’t ready to measure their patients’ outcomes, encourage them and make it easy for them to start.

For Administrators in Payer Organizations One of the easiest ways to identify a cohort or segment of people whose outcomes can be improved is to analyze your claims data by member and identify those whose costs are relatively high. Cost won’t be the only

https://doi.org/10.1017/9781009240925.005 Published online by Cambridge University Press

20

How to Measure Health Outcomes

information you use, but it is often a first step. Higher costs can be a signal of ineffective and inefficient care, resulting in less-than-optimal health outcomes for your members. Once you have identified the people with higher costs, you could determine the medical conditions that they have. Those medical conditions become your cohorts that you can then refine further into segments. To determine which cohorts and segments to focus on, identify conditions for which care is highly variable. Wide variation in care is often unwarranted; it is not based on the latest science or differences among patients. Unfortunately, unwarranted variation is well documented in every country, every region, and for every medical condition that researchers have studied.1,2 Variation in outcomes can be a signal of a gap between what medicine knows how to achieve and what is happening in practice. Addressing these variations requires partnering with clinicians in your network who take care of these patients. Identifying variation in care in this manner also applies to administrators working in health care provider organizations who have access to enterprise-wide data. As you consider selecting a cohort, look for opportunities to improve value dramatically by achieving better outcomes and lower costs. This might mean that your initial focus isn’t on the most expensive patients. It could be the case that care for people with chronic conditions offers more immediate opportunity for dramatic outcome improvement than extremely expensive care for trauma or transplant patients, for example. An alternative starting point is to identify which patient segments require an interdisciplinary care team. Given the fragmentation of most care today, you may be able to make immediate and considerable improvements by integrating care delivery. The point is to look for opportunities to improve outcomes, not just places to reduce costs. Once you have identified a cohort of interest, share the data that you have with the health care provider organization and clinicians who primarily serve the people in that cohort. An opportunity can arise that brings together the clinicians, health care administrator, and payer to improve patient outcomes. Take, for example, the German health plan, Die Kaufmännische (KKH), which used three criteria to identify conditions: high spending, the need for multidisciplinary care, and evidence that there is established good practice that was not being consistently applied. With these criteria, they identified people suffering from migraines as a potential segment of patients who could have better outcomes at lower costs. Leaders of the health plan approached a local physician with expertise in migraine care and the University Hospital of Essen in Essen, Germany with the idea to create an integrated practice unit for people suffering from migraines. The clinical team designed the care and tracked patient outcomes, which improved remarkably. Over time, the overall costs for patients also declined, making the improvement in value unmistakable.3

https://doi.org/10.1017/9781009240925.005 Published online by Cambridge University Press

Where to Start?

21

Step 2A: Gather Baseline Data Gathering baseline data on patient outcomes gives you a sense of how your patients within the cohort or segment you identified are either doing now or were doing in the recent past. Some common endpoints to examine include length of hospital stay, time to seeing a specialist, costs, readmissions, emergency room (ER) visits (including visits where patients left without being seen), frequency of diagnostic imaging and/or laboratory testing, and mortality. Aside from mortality, these measures are mostly about the inputs of health care delivery and not the output; however, they can give you quick insight into your performance. Additionally, patients care about the impact of these measures. In other words, they don’t want hospital stays beyond what’s necessary; they want to be seen as quickly as possible when referred to a specialist; they don’t want unnecessary procedures or tests to find out what is wrong with them; and so on. Each of these potentially adds to the chaos that patients experience, rather than enabling improvement in the outcome of calm. These data will also illuminate any variations that exist in processes and/or costs. So, using these types of endpoints, while not outcomes, can still yield performance insights that can get you one step closer to knowing and improving your patients’ outcomes. Two broad categories of data that can be used for gathering baseline information are administrative and clinical, and they can be found in a variety of sources (Table 3.1). Administrative data are generated by hospitals, clinics, health centers, and insurance companies through the process of delivering and billing for health care services. The clinical validity and accuracy of administrative data is questionable because the primary purpose of these systems is billing; however, the data can be useful in other ways, depending on what you are trying to measure. Clinical data are generated by those caring for patients or the patients themselves through the course of care. Clinical data are derived from patients’ medical records, clinical notes, laboratory and diagnostic imaging reports, and patient-reported outcome questionnaires/surveys. Each data source offers different types of information, so it is important to understand the advantages and disadvantages of each when analyzing baseline data. You should also be careful to ensure the quality of the data prior to conducting your analysis. We will talk more about data quality in Chapter 6. Baseline data can be collected in two ways: by analyzing existing historical data (known as retrospective or historical analysis) or by gathering new data (known as prospective analysis). The method you choose determines where to look for the data you need. More on each of these methods is in the following section.

https://doi.org/10.1017/9781009240925.005 Published online by Cambridge University Press

https://doi.org/10.1017/9781009240925.005 Published online by Cambridge University Press

Disadvantages

• Questionable clinical accuracy and validity • Limited clinical details • Claims data are tracked by member number instead of by person. If a person loses coverage and regains it, a new member number is generated. As a result, that person may now be doublecounted, or they can no longer be followed long-term

Advantages

• Readily accessible • Offers individual patientlevel data • Claims data from payers include data from different sites of care, enabling a more longitudinal picture of care delivery

Where to find it

• Provider billing systems • Scheduling systems • Financial decision support systems • Payer claims systems

Examples

• Number of patient encounters • Encounter location • Length of stay • Discharge status (alive/dead) • Number of readmissions • Diagnosis and procedures using billing codes • Diagnostic tests • Prescribed medications • Costs • Charges • Reimbursements

Data type

Administrative

Table 3.1 Different types of data

https://doi.org/10.1017/9781009240925.005 Published online by Cambridge University Press

Clinical

• Vital signs • Pain scores • Diagnosis and procedures documented by clinical care providers • Laboratory values • Imaging reports • Mortality • Timestamps • Patient-reported outcomes (PROs) and quality-of-life (QOL) surveys

• Electronic medical record • Laboratory information systems • Diagnostic imaging information systems • Pharmacy information systems • PROs and QOL surveys that provide information directly from the patient

• Contains clinical details needed for clinical risk stratification • Increased clinical validity

• Completeness of clinician documentation can vary • Mistakes in clinical documentation can be perpetuated over time in electronic health records because of the copy/ paste function • Difficult to access clinical data because how data is documented and displayed in the medical record is often not stored in a format that enables data to be retrieved easily for analysis; accessing clinical data largely depends on the analytic capability of the organization

24

How to Measure Health Outcomes

Retrospective or Historical Analysis If you have historical data, pick a period of time in the past to study (Figure 3.2). How far back to go will depend on a few factors: 1. how much data is available; 2. how easily you can do the analysis (larger data sets can take longer to analyze depending on the quality of the data, which we’ll discuss in Chapter 6); and 3. whether there is a particular historical milestone in the clinical program or organization that would be a natural starting point for your analysis (e.g., when a new clinical or health program started). In most cases, however, going back at least one year is probably sufficient.

Requesting and Analyzing Retrospective Data Whichever data type you have access to and choose to use, remember that when you do your analysis, the unit of analysis must be the at the level of the individual person. Using a deidentified, previously aggregated data set will not be useful if your goal is to identify gaps in outcomes for the individuals you serve. Many types of health services research require aggregated data sets, but those types of analyses have a different purpose and focus. For example, they may show population-level effects of social determinants of health or the impact of hospital-wide initiatives on handwashing or other safety measures. They do not show the change in outcomes of the individual people you serve. It is not possible to identify individuals in deidentified, aggregated data. Once you have identified which data source to use, you will need to think about the questions that you want to answer with your data. Here are some examples: 1. How many people have presented to the ER with extreme pain within the last six months? 2. For those people, how long did it take before the pain was manageable and patients were able to be discharged home? 3. Were any of them readmitted for the pain within a week? 4. How many people presented to the ER for any reason? 5. Are they the same people? 6. Is there a pattern?

End

March 31, 2021

Start

April 1, 2022

Figure 3.2 Example of a timeframe for a retrospective or historical analysis.

https://doi.org/10.1017/9781009240925.005 Published online by Cambridge University Press

Where to Start?

25

If the data you need for your retrospective analysis already exists and you work in a larger health care organization, you will likely be working with a report writer or analyst who can create a query to obtain the data that you need in order to answer your questions from the appropriate data system. If so, it is very important that you describe your data request clearly and with as much specificity as possible. Sometimes the data that you need doesn’t exist in a retrievable or analyzable form. If this is the case, you may need to manually review medical records or other information systems to find the data that you need and put them into a database. Research nurses, medical students, and/or other health profession students can support with gathering and entering the data. A database is a structured, systematic collection of data. For purposes of analyzing individual patient outcomes, this database should be organized as a relational database. A relational database means that the database is organized into rows and columns whereby each row represents a single record and each column represents an attribute of the data in that row or record. The data in the rows and columns are related to each other. Chapters 5 and 6 will go deeper into creating basic databases and analyzing data.

Prospective Data Analysis Another way to collect baseline data is by measuring forward to a future date, for example, from today through the next three months. This is referred to as prospective data collection (Figure 3.3). It is often easier and less resource intensive to collect data prospectively than it is to retroactively gather historical data unless you have research staff or a data analyst who can readily collect retrospective data for you. Let’s return to the example in Chapter 1 of the group of pediatric psychologists who wanted to improve the outcomes of their patients with obsessive compulsive disorder (OCD). They had an idea about the outcomes that they wanted to track and collected data for roughly three months. When the data revealed tremendous variation in outcomes achieved by different care providers, leadership introduced several improvement initiatives. If you choose to conduct a prospective data analysis, you will need to create a database. The term registry is sometimes used interchangeably with the word database; however, registries and databases are different. Registries are Start

April 1, 2022

End

June 30, 2022

Figure 3.3 Example of a timeframe for a prospective analysis.

https://doi.org/10.1017/9781009240925.005 Published online by Cambridge University Press

26

How to Measure Health Outcomes

databases designed using observational research methodology to collect data on treatment and clinical outcomes over time for a particular condition. Creating a clinical data registry requires significant human and data management resources – more than you’ll likely have or require as you launch your initial measurement efforts. We will discuss registries more in Chapters 4 and 5. For the purposes of just getting started with prospectively tracking your outcome data for a finite period of time, I recommend creating a simple relational database. I will go into detail on how to do this in Chapter 5.

Step 2B: Identifying the Outcomes that Matter Most to People One way to understand the outcomes that matter most to patients is by defining what success looks like to them. What do they see as standing in the way of their achieving the health that they want? Again, returning to the example in Chapter 1 of the group of pediatric psychologists who wanted to improve the outcomes of their patients with OCD, success was defined by the children being able to focus on school and play with friends without the OCD disrupting these experiences. If you are a clinician, sometimes it is possible to begin to understand the outcomes that matter most to your patients based on the questions that they or their families ask you. Take, for example, the parents of a child born with cleft lip and palate4 who are discussing surgical repair with a surgeon. The parents may ask, “Will my child be able to speak normally later in life?” or “With such a prominent scar on their face, will my child be bullied?” Paying attention to questions like these provides some insight into the top concerns of patients and/or their families. For those who work in health care administration, whether on the provider organization side or the payer side, these types of conversations may not occur as readily, but there is a way to have them. One way for anyone working in health care to learn the outcomes that matter most to the people that they serve is to conduct Experience Group™ sessions. The Experience Group™ methodology uses qualitative research techniques in order to understand the lived experiences of people who share a set of health circumstances – that is, who are part of the same segment. Through small, lightly moderated group discussions, unmet health needs surface and the outcomes that matter most to patients and families are identified.5 These sessions typically elucidate a manageable number of outcomes (three to seven) that are meaningful to most patients within your identified segment. More information about how to conduct Experience Group™ sessions can be found in the Experience Group Handbook (S. Wallace, E. Teisberg. Experience Group Research Methodology: A Handbook for Understanding Outcomes that Matter Most to Patients. Unpublished, 2019).

https://doi.org/10.1017/9781009240925.005 Published online by Cambridge University Press

Where to Start?

27

As you begin to identify the outcomes that matter, organize them in the Capability, Comfort, and Calm outcome measurement framework. When people receive care from you or your organization, are their functional outcomes restored to the greatest extent possible? Are pain, anxiety, and emotional suffering reduced? Is care delivered in an accessible, clear, and affordable way that enables daily life to continue as smoothly as possible?

Action Steps ▯ Identify potential patient cohorts or segments whose outcomes you could measure. ▯ Identify clinicians who you can support who have expressed a desire to measure outcomes. ▯ Conduct Experience Group™ sessions. ▯ Define what success looks like for the people that you serve. ▯ Identify the outcomes that matter most. Jot down your ideas below.

Having identified the outcomes that matter to the people that you serve, the next step is to find measures that you can use to track progress in improving those outcomes. Let’s continue to look at this in the next chapter, “Identifying Outcome Measures.”

References 1. J. E. Wennberg. Tracking Medicine: A Researcher’s Quest to Understand Health Care. Oxford University Press, New York, 2010. 2. E. S. Fisher, D. E. Wennberg, T. A. Stukel et al. The Implications of Regional Variations in Medicare Spending. Part 1: The Content, Quality, and Accessibility of Care. Ann Intern Med 2003; 138: 273.

https://doi.org/10.1017/9781009240925.005 Published online by Cambridge University Press

28

How to Measure Health Outcomes

3. M. E. Porter, C. Guth, E. M. Dannemiller. The West German Headache Center: Integrated Migraine Care. Harvard Business School Publishing, Cambridge, MA, 2007. 4. Facts about Cleft Lip and Cleft Palate. Centers for Disease Control and Prevention. www.cdc.gov/ncbddd/birthdefects/cleftlip.html. Accessed August 16, 2022. 5. R. E. Silverman. When Patients Share Stories, Health Insights Emerge. The Wall Street Journal. www.wsj.com/articles/when-patients-share-stories-health-insightsemerge-1488164580. Updated February 26, 2017. Accessed August 16, 2022.

https://doi.org/10.1017/9781009240925.005 Published online by Cambridge University Press

Chapter

4

Identifying Outcome Measures

As described in Chapter 1, the outcomes that matter most to people fall into three domains – Capability, Comfort, and Calm. Once you identify the outcomes that matter most to the people that you serve, the next step is to identify measures that you can use in order to see if people are improving with respect to those outcomes. The Capability, Comfort, and Calm outcome measure taxonomy (Figure 4.1) illustrates the relationship between outcomes and outcome measures. It can be a helpful tool when organizing your outcome measure set. For example, if you determine that an outcome that matters to the individuals that you serve is to walk without pain, you will then look for measures that capture progress toward reducing knee pain. One way to measure that outcome is the Knee Injury and Osteoarthritis Outcome Score (KOOS),1 which assesses how often a person experiences knee pain and how their pain impacts their ability to function. From there, you can incorporate additional measures if needed. Outcome measures may be expressed as a single question like, “Is the person alive or dead?” They may also be defined by scores calculated from questionnaires that are administered to patients. In recent years, patient-reported outcome (PRO) surveys have become popular as clinicians try harder to assess the outcomes that people want to achieve from care, not just the results of interest to clinicians or payers. These surveys were developed to get the patient’s perspective on the outcomes of clinical treatment. They are standardized, validated questionnaires that allow for scientific comparison across different treatment modalities. (Validated means that the questionnaire has been systematically evaluated in order to ensure that it produces true and consistent results and, for a screening questionnaire, is sensitive enough to correctly identify that a patient may have a particular condition that warrants further evaluation.) Many validated PRO surveys are available for free, though some require a licensing fee. I discuss several resources for identifying PROs in this chapter. Any approach to identifying outcome measures requires clinician leadership. Clinicians are best positioned to know which measures show achievement of the outcomes that are relevant to individuals, so if you are a health care administrator, make sure to include them in this process. 29

https://doi.org/10.1017/9781009240925.006 Published online by Cambridge University Press

https://doi.org/10.1017/9781009240925.006 Published online by Cambridge University Press

Calm

Comfort

Additional measure

Generalized Anxiety Disorder Screener-7 (GAD-7)

Additional measure

Knee Injury and Osteoarthritis Outcome Score (KOOS)

The outcomes/results that matter to our patients

Additional outcome ...

The measures to show us if we are achieving the results (i.e. success)

Additional measure

Don’t have to miss too much Time missed from work work

Additional outcome ...

Feel less anxious about my prognosis

Additional outcome ...

Figure 4.1 Capability, Comfort, and Calm outcome measure taxonomy.

Outcomes that matter most to patients

Capability

Walk without pain

CAPABILITY, COMFORT, AND CALM OUTCOME MEASURE TAXONOMY

Identifying Outcome Measures

31

To start, do a quick review of the literature to see which outcomes others have tracked for the medical condition or patient segment that you are interested in. It can also be helpful to look at the operational metrics that your organization tracks and consider if there are any that map to Capability, Comfort, and/or Calm. For example, your organization or unit may track emergency room (ER) visits or returns. While those metrics in and of themselves are probably not what matter most to your patients, the disruption to their lives that the ER visit creates, does. This sort of disruption can be categorized and tracked for improvement under the Calm domain. On-time surgery starts, another common operational metric tracked in acute care settings, can similarly be reframed: If patients arrive expecting to have surgery and end up waiting for an extended period of time, calm is disrupted, and anxiety increases. As these examples show, although health care needs to shift its focus from measuring processes to measuring outcomes, certain processes and indicators can still be useful to track improvement in the outcomes that matter most to patients if they are framed correctly: as the means to an end, rather than ends in and of themselves. Another example comes from a primary care physician in Louisiana who treated a woman with diabetes. The physician learned that the outcome that mattered most to her was getting pregnant and having a healthy baby. She sought fertility treatment but was not eligible for it until her HbA1c level was less than 8%. Once the physician understood this, he explained why achieving lower HbA1c levels was important to her meeting her goal. From there, the physician and patient worked together to lower her HbA1c levels, and she ultimately conceived and had a healthy baby (R. Lee, A. Madore, K. Carberry. The Comprehensive Care Center at Baton Rouge General Medical Center. Unpublished case, Value Institute for Health and Care at the University of Texas at Austin, 2020). So again, you may not have to search far for a measure; sometimes you can just think differently about the ones that you currently use. For many conditions, you can also use publicly available collections of measures identified by organizations such as the International Consortium for Health Outcomes Measurement (ICHOM) and HealthMeasures. An organization committed to defining and standardizing outcome measure sets that reflect the outcomes that matter most to patients, ICHOM develops measure sets through a consensus-based process by convening an international group of clinicians with expertise in a particular medical condition and patients who have that medical condition. The group does not create measures; instead, it develops a consensus around which existing validated instruments can be used to measure meaningful patient outcomes. The organization has created more than 30 standard sets to date that are available to the public.2 These sets are intended to represent the minimal standard set of outcomes to measure for a particular condition.

https://doi.org/10.1017/9781009240925.006 Published online by Cambridge University Press

32

How to Measure Health Outcomes

Implementing every measure in the standard set, many of which are indepth questionnaires, can seem overwhelming if you are just starting. So, instead of trying to implement all of the measures at once, pick one to start and add measures as you can. HealthMeasures, also known as the Person-Centered Assessment Resource, is funded by the National Institutes of Health (NIH) in the USA to provide an online source of evidence-based, psychometrically sound outcome measures, many of which are available in Spanish and other languages. On its website, you can search for measures, download instruments for free, and access guidance for scoring and implementation,3 including a list of data collection platforms that have the HealthMeasures surveys built in, such as REDCap and the electronic health record produced by Epic Systems (Madison, WI). The website also offers case studies of organizations that have implemented the HealthMeasures questionnaires. One of the most popular measurement systems supported by HealthMeasures is the Patient-Reported Outcomes Measurement Information System (PROMIS), a collection of over 300 measures of health for adults and children.4 PROMIS Global-10 is a frequently used general measure of health and well-being. Clinical registries are also a source of potential measures. As mentioned in Chapter 3, registries are databases designed using observational research methodology to collect data on treatment and clinical outcomes over time for a particular medical condition. The organizations that support registries often publish annual reports with outcome data. Reviewing these reports can provide ideas of outcome measures that may be relevant to you. Although not comprehensive, the NIH maintains a list of many national registries with hyperlinks to more information about each.5 You can also develop new measures, but the methods for doing so go beyond the focus of this book. Developing measures takes time, and there is literally a science to it. Before you start down the path to develop a new measure, first ask yourself if it’s really necessary in order to achieve your outcome measurement goal and confirm that there aren’t adequate existing measures. If you can develop a new measure efficiently and it will truly help you help your people more, then it may be worth it. If you are considering this option, a great reference to get started is a 2018 article entitled “Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer.”6

Refining Your List of Outcome Measures Once you start searching for measures and beginning your list, you can easily feel overwhelmed by the number of measures you or your team may want to use. It is tempting to reason that the more information you have about how patients are doing, the better you can help them manage their health.

https://doi.org/10.1017/9781009240925.006 Published online by Cambridge University Press

Identifying Outcome Measures

33

However, clinicians need to be able to respond to data in order to improve their care of patients. The process of collecting and analyzing outcome measures should not become so overwhelming that it is impossible to learn and act in a timely fashion. The goal is to find a balance between considering too little and trying to analyze too much. While you may initially come up with a long wish list of metrics that you could track given unlimited time and resources, you ultimately need to narrow that list down so that is manageable for your practice, team, or organization. As you prune the list, do a simple check on sufficiency of what you’ll learn: Will the measures that you’ve chosen tell you if the patient is getting meaningful help from the care being given? A focused set of measures is key to successfully launching a sustainable outcome measurement program. Remember that you can always add measures over time as you become more experienced with outcome tracking. When I started measuring outcomes for congenital heart surgery, the national registry for congenital heart surgery had a “minimal data set” that participants had to complete on each patient for the record to count as complete. This minimal data set included basic demographic information, diagnosis and procedure information, and hospital discharge status – alive or dead. As programs became more efficient at collecting data, the national registry database team asked that additional data be submitted to provide more clinical details for patient risk stratification, including preexisting medical conditions and more information about in-hospital complications.

A Few Guiding Rules Can Help You Narrow Your Measure Set First, keep your Capability, Comfort, and Calm measure set to a reasonable size. While there is no “correct” number of measures to aim for, you may want to start relatively small, particularly if you are new to outcomes tracking. Choose the minimum necessary to determine success for your patients. Around four to five measures are often enough to capture a significant amount of the information you need. If that feels like too much, remember that tracking even one meaningful measure is better than none. Second, make sure that all of the measures that are included on your list actually feed back to your goal of improving outcomes in a meaningful way. Will the information provided by the measure directly impact your ability to improve patients’ health in a way that matters to them? Is it clinically actionable and thus able to affect the care or treatment of patients? If the information will be simply “nice to have” and not directly actionable, it is not worth your time to collect. Third, consider your ability to collect accurate and reliable data for the measures. Will data need to be tracked by hand and, if so, do you have the staff

https://doi.org/10.1017/9781009240925.006 Published online by Cambridge University Press

34

How to Measure Health Outcomes

time and resources to do so? What burden, if any, will be put on patients to provide this data, and are those expectations reasonable? If collecting robust data for this metric will be too onerous, it is not worth including; you can table the measure for now and revisit it at a later date when there is a more feasible way to get the information. (We’ll look more closely at the feasibility of collecting reliable and accurate outcomes data in Chapter 5.) If you are working with a team to narrow your list of measures, a common method for doing this is the Modified Delphi method. This is a consensusbased process that consists of a series of voting rounds among clinical experts – and ideally patients – on an initial list of measures until the number of measures is tapered to a manageable, preidentified number. Keep in mind that some measures are more extensive and complex than others. Even if you narrow your selection to one measure, if that measure is a long questionnaire that leaves patients feeling fatigued and providers with more information than they need to make clinical care decisions, it may not be the best measure. An example of how to tackle this problem comes from the psychologists at Cincinnati Children’s (Chapter 1) who found a way to track session-by-session if children with obsessive compulsive disorder (OCD) were making progress in treatment. The psychologists created a short version of the Children’s Yale–Brown Obsessive Compulsive Scale (CY–BOCS), a standard instrument used in the diagnosis and treatment of OCD. The CY–BOCS took approximately 45 minutes to complete, so it was not feasible to administer it at every treatment session. Through a consensus-based method, the psychologists extrapolated four key questions from the instrument to be used in each session and reserved the full instrument for the beginning and end of treatment. When treatment was completed and subclinical status achieved, both the CY–BOCs and the shortened version showed improved results. This is an example of a situation where it may be beneficial to adapt an existing measure to better meet your outcomes measurement needs. Other groups are attempting to do something similar with the KOOS survey by administering a single question, instead of the entire questionnaire, to ease survey burden on patients and generate actionable data that can inform treatment decisions readily in the clinical setting.7 With your outcome measure set created, the next step is to clearly define your measures so that you can begin collecting data. We’ll examine how to do that in the next chapter.

Action Steps ▯ Conduct a literature and/or web search for outcome measures for the cohort or segment that you identified. Which of these measures align with the outcomes that you identified in Chapter 3?

https://doi.org/10.1017/9781009240925.006 Published online by Cambridge University Press

Identifying Outcome Measures

35

▯ Find out which measures your organization currently collects. Can any of them be reframed as Capability, Comfort, or Calm? ▯ Explore the standard sets on ICHOM’s website. Study the measures within them and identify ones that you can use. ▯ Explore the HealthMeasures website for possible measures to use. ▯ Explore the NIH list of registries; visit the registries that are relevant to your work. ▯ Keep a list of possible outcome measures that you can use and organize them using the Capability, Comfort, and Calm outcome measure taxonomy. ▯ Begin to refine the list of outcome measures that you have identified.

References 1. KOOS Knee Survey. www.koos.nu/koos-english.pdf. Accessed August 16, 2022. 2. ICHOM Standard Sets. www.ichom.org/standard-sets/. Accessed August 16, 2022. 3. HealthMeasures. www.healthmeasures.net/index.php. Accessed August 16, 2022. 4. PROMIS Health Organization. www.promishealth.org/. Accessed August 16, 2022. 5. List of Registries. National Institutes of Health (NIH). www.nih.gov/healthinformation/nih-clinical-research-trials-you/list-registries. Accessed August 16, 2022. 6. G. O. Boateng, T. B. Neilands, E. A. Frongillo, H. R. Melgar-Quiñonez, S. L. Young. Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer. Front Public Health 2018; 6: 1–18. 7. D. C. Austin, M. T. Torchia, P. M. Werth et al. A One-Question Patient-Reported Outcome Measure Is Comparable to Multiple-Question Measures in Total Knee Arthroplasty Patients. J Arthroplasty 2019; 34: 2937–2943.

https://doi.org/10.1017/9781009240925.006 Published online by Cambridge University Press

Chapter

5

Collecting and Analyzing Outcome Data

This chapter will focus on how to collect and analyze your outcome data. It will focus on collecting the data that you need in order to know your patients’ outcomes. It’s important to note that the data that you need may not be all of the data that you want. I can’t tell you how many times I have seen outcome measurement efforts stall because clinicians want to design a database that captures the answer to every question that they have about a particular medical condition. That’s typically an overwhelming task, especially when you are just getting started. I once worked with a team of physicians who were establishing a new clinical program for patients with a specific medical condition and wanted to build a database to track outcomes. One physician in the group had already started using a Microsoft Excel spreadsheet to track a small number of outcomes. Others in the group were more ambitious and wanted to collect a substantial amount of diagnostic data from every clinical visit for future research. The scope of the effort grew so large that it took too much time and the database never materialized. To this day, however, the physician who started the Excel spreadsheet is still using it and has accumulated over three years’ worth of outcome data. To be clear, collecting data to answer research questions is important; it is just not the same primary purpose as tracking outcomes, and can become a barrier to getting started. The lesson is to keep it simple and just start measuring. It is also important to remember that the reason that you are collecting outcome data is to know if your patients are getting better, worse, or staying the same, not to create an exhaustive data repository for future research. This distinction matters because when you track outcome data with the primary intention of conducting research, you typically collect as many variables as possible in order to answer future research questions that you may not even have thought about yet. Outcome tracking shares some of the same measurement principles as outcome research, but the intention is different. While the data that you accumulate from outcome tracking may be used for future research, outcome tracking has the explicit purpose of measuring how the patient’s outcomes are improving through the course of care. 36

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

Collecting and Analyzing Outcome Data

COLLECT & STORE

INTEGRATE & VALIDATE

EXTRACT & ANALYZE

37

SHARE & IMPROVE

Figure 5.1 Key steps in the data management process.

The next several sections describe the mechanics of how to begin collecting data for patient outcome measurement. The primary audience for this chapter is individual clinicians who want to start measuring outcomes in their clinical practice. The next chapter, “Scaling Outcome Measurement,” will address the role of health care administrators in the data management functions of outcome measurement. This chapter of the how-to guide gets a little more technical; however, my goal is to write at a level that is accessible to people without a technical background. The intent is not to make you an expert at database creation but to provide you with enough knowledge to have informed conversations with colleagues and partners who are.

Foundational Functions of Data Management As you begin thinking about how to collect your patient outcome data, you will need to understand a few foundational functions related to data management. Data management refers to how data are collected, stored, validated, analyzed, and shared (Figure 5.1). Every data system, from the simplest to the most complex, involves these basic functions. Understanding these functions is critical to creating an outcome tracking system that produces valid and reliable information in order to drive improvement.

Collecting, Recording, and Storing Data As a first step, you need a way to collect, record, and store outcomes data. Before discussing how to do that, I want to say a few words about data integrity. Data integrity refers to the accuracy, validity, and reliability of data (Figure 5.2). How data are collected directly impacts data integrity. This is where the familiar saying “garbage in, garbage out” applies. Having accurate, valid, and reliable data is critical in outcome measurement. Not enough can be said about its importance and the steps and processes necessary to ensure it, as discussed later. Questionable data integrity can contribute to the dissemination of inaccurate or incomplete information, increase clinician

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

38

How to Measure Health Outcomes

Aspects of Data Integrity 1.

Accuracy: How well the observed value of a measurement matches the true value

2.

Validity: How well the data measure what the observer intended to measure

3.

Reliability: How consistently a value is measured as it was

Figure 5.2 Aspects of data integrity.

distrust of data, and potentially misguide care decisions. Keeping data integrity at the top of your mind when beginning to gather outcome data is essential. Data may be gathered from an existing data repository or by direct patient observation and inquiry. Once collected, the data must be recorded and stored for future reference and analysis. At its simplest, this can be accomplished with paper and pencil. For example, Ernest Codman, a surgeon who practiced in the early 1900s, collected outcome data on cards for each of his patients. He believed that every patient should be followed long enough to know whether the treatment was successful or not.1 Today, we have the technology to make data collection less tedious. Nonetheless, if that technology becomes a barrier to getting started, don’t hesitate to just start with paper and pencil, creating a ledger of sorts to record and track each of your patients’ outcomes. It is common for patient questionnaires to start on paper before moving to an electronic device like a tablet or iPad. If you are familiar with a spreadsheet program like Excel or a database program like Microsoft Access or REDCap (Research Electronic Data Capture), you can get started with one of these tools. Building a database from scratch requires more specific information technology (IT) training and expertise than building an Excel spreadsheet. Database programs are relatively easy to acquire; however, they require specific technical knowledge and skills that many clinicians do not have. Hiring a database developer can solve this problem; however, you may not have funding for such a position. This is where REDCap becomes an attractive database option because the costs to use it are minimal. I provide more details on REDCap later in this chapter.

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

Collecting and Analyzing Outcome Data

39

Excel and REDCap by no means represent an exhaustive list of data collection options. They simply are easier and less expensive ways to get started. As a result, we’ll focus just on those two options in this chapter. Let’s start by looking at Excel.

Using Spreadsheets to Collect Your Outcome Data Excel is essentially an electronic ledger, or list, designed to tabulate and calculate values stored in cells. When using Excel spreadsheets to track outcomes, you should organize your data into rows and columns, whereby each row contains a unique record about a single subject – in our case, a patient – and attributes related to that single subject are labeled in column headers. The cell at the intersection of the row and column contains the value of a specific attribute for that patient. Column headers should include the following basic demographic information: • last name • first name • preferred name • date of birth (DOB) • address (or, at a minimum, zip code) • sex, gender, race, and ethnicity, which are critical in order to identify and address any disparities in outcomes that exist in relation to these identity characteristics • relevant social and economic data that may influence health (e.g., education level, insurance type) • unique patient identifier • record identification number (record ID) One of the most important attributes of the individual patient record is a unique patient identifier, such as a medical record number (MRN), which is assigned to a patient when they see a health care provider. This unique patient identifier should be recorded with every record. You will also need to establish a record ID number. The record ID is just what it sounds like – a number associated with each single record that you are creating. Unlike the patient identifier, the record ID is typically tied to an event such as a clinic visit, therapy session, hospital admission, or surgery. Most of the demographic information can be gathered from administrative data generated by hospitals, clinics, health centers, and insurance companies through the process of billing for health care services. Data easily collected from administrative sources include name, DOB, address, and insurance type.

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

40

How to Measure Health Outcomes

Be cautious, however, about the validity and reliability of race, ethnicity, and gender data gathered from administrative sources. Often, patients are not asked to provide this information but rather the person performing the patient registration process makes an assumption based on how the patient presents and puts that information into the patient’s record. To ensure that race, ethnicity, and gender data are accurate, it is best to ask the patient how they identify and record that information. Knowing that racism contributes to health disparities, it is necessary that we collect data on race in order to analyze outcomes in the context of this variable, identify any disparities, and address them. Column headers should also include the names of the other attributes that you plan to track, such as relevant health conditions that may influence outcomes, and the outcome measures themselves (e.g., KOOS scores). With respect to a person’s medical condition(s) and other clinical factors that influence outcomes, only clinical data, not administrative data, will provide the most valid and reliable information. For example, in congenital heart surgery, different operations are assigned risk scores so that outcomes can be risk-stratified during analysis. Those risk scores are not found in administrative data that are generated through billing and are critical to understanding outcomes. Outcome data must come from clinical sources, which, as discussed in Chapter 3, provide data that are generated by those caring for patients or the patients themselves. You will enter a new record each time that the patient is seen. Your spreadsheet may have multiple records (rows) for the same patient if they have multiple visits, and those visits may be related to the same or different medical conditions. Sometimes visits are referred to as encounters. For each of these records, the record ID will be different, but the unique patient identifier will stay the same. REDCap, as I will discuss later, generates new records in a different way. When using Excel to collect your outcome data, you will need to create a data dictionary. For our purposes, I define the term data dictionary in the simplest way – a list of the variables used in your database, their definitions, and where you are getting each variable (i.e., the sources of your data). Your data dictionary should include a definition for every attribute listed in the columns of your spreadsheet. For example, if the attribute is readmission to the hospital, you need to specify in your data dictionary how you are defining readmission. Is it readmission after a certain timeframe following discharge? Are you looking for all causes of readmission or causes related to the hospitalization or procedure? These types of questions should be answered prior to starting data collection. A data dictionary helps standardize the data collection in order to allow for accurate analysis later, and is one way to ensure data integrity. Whether using Excel or REDCap to track patient outcomes, it is critical to remember that the unit of analysis is the individual patient and not a health

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

Collecting and Analyzing Outcome Data

41

care event like the clinic visit or hospitalization. In other words, when you are tracking patient outcomes, you are collecting and analyzing data that directly tells you something about the health outcomes of the patient, not what happened on a particular day in clinic to the patient without regard for the health outcomes. To illustrate, I’ve seen patient “outcome” databases created to track if a screening questionnaire was given to patients when they came to clinic, yes or no. Whether or not a patient received a screening questionnaire is not a health outcome, nor is it an outcome that matters to patients. To turn that measure into a health outcome measure would be to track if the symptoms described on the screening questionnaire were improving or not over time for each individual patient. Data about what happened in clinic may be collected for operational purposes, but that should not be confused with measuring patient outcomes. Table 5.1 is an example of an Excel spreadsheet organized to track outcomes. Recall the cardiologist mentioned in Chapter 2 who wants to track side effects from statins. These side effects may include muscle pain, drowsiness, or memory loss. If the patient has more than one side effect, the spreadsheet will need additional columns to track each of those outcomes separately (e.g., side effect 1, side effect 2, etc.). Spreadsheets are fairly intuitive to build. One can simply open Excel on a computer, create lists of patients with the attributes to measure, and start tracking within a few hours. For this reason, many people like to begin tracking outcomes with Excel, and it is a very reasonable option for getting started. Excel is typically available as standard software on PCs in work environments or can be installed with help from your organization’s IT department.

Using REDCap to Collect Your Outcome Data Spreadsheet functionality can quickly become limited, however, when trying to track outcomes. While spreadsheets have a capacity of over a million rows of data, they are not designed to reliably store and organize large amounts of data. Collecting large amounts of outcome information can quickly become unruly. If a spreadsheet is not filtered or sorted correctly, for example, data integrity can come into question because the values associated between rows and columns may no longer match. Also, once large amounts of data are stored in Excel, executing data analysis functions becomes extremely slow. It is also difficult to create data entry standards using Excel, so how data are recorded becomes user-dependent. If you have more than one person entering data, there is a good chance that how data are recorded will vary, impacting data integrity down the road.2 Bearing in mind these limitations of Excel, consider using REDCap database software as a more durable option. It is a web-based application designed for people who don’t have technical backgrounds to be able to create and

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

Last name

Smith

Jones

Smith

Record ID

001

002

003

Oscar

Jill

Oscar

First name

12345

15643

12345

MRN

5/6/ 1941

1/7/ 1949

5/6/ 1941

DOB

01/ 12/ 2019

Date of visit

Medication

Atorvastatin

Primary diagnosis

Hyperlipidemia

Muscle pain

Side effect1

Table 5.1 An example of how to organize outcomes data collection using a spreadsheet (data are fictional)

Memory loss

Side effect 2

Collecting and Analyzing Outcome Data

43

manage surveys and databases. In other words, it allows you to build a database without hiring a database developer. As described on their website, REDCap is a secure, web-based software platform designed to support data capture for research studies, providing 1. an intuitive interface for validated data capture; 2. audit trails for tracking data manipulation and export procedures; 3. automated export procedures for seamless data downloads to common statistical packages; and 4. procedures for data integration and interoperability with external sources.3,4 The start-up time for developing a REDCap database is longer than it is with Excel, and some training is required; however, the upfront time lays a solid foundation for your outcome tracking efforts, and with practice, using REDCap becomes easier. Another helpful feature of REDCap is that it can be used by multiple users at multiple sites easily. This becomes more important if you plan to grow your outcome tracking database into an outcome research database that will enable the comparison of outcomes between different clinical teams at the same location or across different sites. One of REDCap’s biggest benefits is that it guides the user in each step of the survey or database design process through a series of prompts and options. Although users can design a survey or database from scratch if they prefer, REDCap also includes several templates – for single surveys, multiple surveys for longitudinal use, classic databases, longitudinal databases, and more – so that users don’t have to start from a blank page. Users can pick a template that matches their needs or comes close, then adapt it for their specific use. I’ll illustrate some features of REDCap in the context of selecting a classic database template for your outcome database. One REDCap feature that is particularly useful for tracking outcomes is access to a library of existing data collection instruments and surveys. For example, if you wanted to create a database in order to track outcomes of men diagnosed with prostate cancer, you would select the classic database template, then search the REDCap Shared Library using the key words “prostate cancer.” REDCap will retrieve several different instruments, and from there you can select the one(s) that you want to use and import it into your database. Once you have done this, the instrument(s) are listed along with the other standard instruments where data can be collected for each patient (Figure 5.3). From here, you can select any of the instruments and make modifications as needed (Figure 5.4) for the instrument modification page. The instruments in REDCap can also be thought of as forms to be completed. Note that a record ID (“study ID” in REDCap) is automatically generated as a field. Once the instruments are ready for data entry, users enter data into new records, as shown in Figure 5.5. With standardized forms, REDCap can more easily maintain data integrity across multiple users. REDCap also automatically generates a data dictionary that includes a list of the attributes/variables

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

44

How to Measure Health Outcomes

Figure 5.3 Screenshot of a list of data collection instruments in REDCap’s Online Designer tool. Used with permission from REDCap.

used in the database as well as technical details like the names of the fields used, field type (text, check-box, radio button), and any branching logic (if this value, then this choice appears). The REDCap data dictionary does not automatically create definitions of the attributes, so you must do this on your own. Data definitions can be recorded in the Field Annotation box using the Online Designer tool when you are creating your data collection instruments/forms. REDCap also provides access to an international consortium of users and experts in database development. The application is regularly updated with new features and new instruments in its library. There are also two mobile applications that enable data to be collected offline at the point of care as needed. REDCap is relatively inexpensive. If the software is already used at your institution, getting started is simply a matter of getting access to the system; if it is not, installing the REDCap application is free, although the institution may need to allocate some IT resources to installation, maintenance, and ongoing user support. If you don’t have access to IT resources, REDCap outlines other options to get started on its website.

More on Choosing to Use Excel versus REDCap Whether you choose to start measuring outcomes with Excel or REDCap depends on how quickly you want to get started and which option seems initially easier. Those with experience using spreadsheets and/or databases

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

Collecting and Analyzing Outcome Data

45

Figure 5.4 Screenshot of instrument modification page in REDCap’s Online Designer tool. Used with permission from REDCap.

may feel comfortable going straight to REDCap. For those with less experience and confidence, Excel provides a relatively low-tech opportunity to practice gathering and organizing data. If you choose to start in Excel, I encourage you to transition to REDCap or another database tool such as Microsoft Access as soon as you can considering the limitations with Excel outlined earlier in this chapter. Excel spreadsheets can be converted into a REDCap database at a later time with some effort. One important note about data collection: As you begin to gather outcome data, it is critical to safeguard your patients’ privacy. The data that you collect are theirs. Keep the data that you collect to the minimum necessary in order to know whether or not outcomes are improving. From a data security standpoint, REDCap is a much more secure option. Users must be authenticated to access the database, and the servers on which REDCap is installed at your local

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

46

How to Measure Health Outcomes

Figure 5.5 Screenshot of Add/Edit Records page in REDCap. Used with permission from REDCap.

institution will have information security safeguards in place for all patient and institutional data. One of the possible dangers of using Excel to track outcomes and house patient data is how easily files can be accessed and emailed by users. Other ethical and legal aspects of measuring outcomes are discussed later in this chapter.

Registries for Outcome Tracking Recall from Chapter 3 that a clinical registry is a database specifically designed using observational research methodology in order to collect data on treatment and clinical outcomes over time. Participating in an established data registry created and maintained by other people or organizations is another way to start collecting your outcome data. Taking this path has the advantage that the heavy lifting of creating a database from scratch is already done; typically, the sponsoring organization has also created a data dictionary and ready-to-use data collection forms. Registry participation often entails a fee, as well as some start-up costs and efforts on the part of your own organization; however, it is much easier to join an existing registry than to create a new one. That said, one important consideration when joining a registry is how readily you will have access to data about your own patients. You should be able to have open access, anytime, to your own data. If the registry is not set up to do that, reconsider joining. In such situations, the primary purpose of the registry is probably amassing as much information as possible for outcome research or quality-improvement studies, not facilitating outcome

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

Collecting and Analyzing Outcome Data

47

improvement at the local level. If participating in a national registry does not allow you to focus on what matters to your patients locally, itis better to begin by measuring with your own tools, as described earlier.

Electronic Health Records for Collecting Outcome Data Ideally, your electronic health record (EHR) system would easily enable patient outcome tracking. After all, it captures clinical data during the course of care and can collect data from patients through portals and remote monitoring devices. Unfortunately, at this point in time, trying to use EHRs to get started with outcome measurement is often a barrier instead of an enabler. I’ve often heard clinicians say, “Why can’t I just build my data collection form into [fill in the blank EHR]?” The simple answer is that existing EHR platforms make building such a database difficult and nearly impossible without the help of the IT department at your institution and its EHR vendor. While efforts are underway to make the transfer of data from EHRs to outcome data registries more seamless,5 these efforts require considerable individual and organizational effort. If you want to move data from the medical record to a registry, you will also need technical assistance from the EHR vendor in addition to your local IT team. This creates a major barrier to getting started, and I have seen many clinicians give up after trying to go down this route.

Additional Considerations for Data Collection Measurement Timeline Before you begin collecting your data, you will need to decide what your “time 0” will be. In other words, will your first record reflect a date in the future or a date in the past? It is easiest to pick a future date and start collecting data prospectively, as described in Chapter 3. If you do need to pick a date in the past and gather historic data through chart review – which is often done when a team wants to understand how patients are doing today – follow the guidance in Chapter 3 for retrospective data gathering. Capturing historic data may require contacting patients whom your practice has not seen for a while. If identifying these patients is critical to achieving your outcome improvement goals, then focus your efforts on trying to reach them and gather the minimal data necessary. A short, scripted survey is a great tool here.

Ethical and Legal Issues When beginning to collect outcome data, have a conversation with each patient about what you are planning to do and why, and make sure that you have their informed consent. This consent should be documented in two ways.

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

48

How to Measure Health Outcomes

The first is by documenting in the medical record that you had a discussion with the patient about tracking outcomes and that they understand and agree. The second is either through a consent-to-treat document or an approved research consent form. Tracking your patient’s outcomes as part of routine care is considered a quality-improvement activity.5 Collecting data for quality-improvement purposes is outlined in the consent-to-treat document that patients are asked to sign when they seek care from a health care provider. The signed consent-to-treat document should fulfill the legal obligation of obtaining consent. However, some institutions may require research consent, as they may consider measuring outcomes to be human subjects research with the intention to contribute to generalizable knowledge. Prior to beginning your outcome data collection, it’s best to consult with your local institutional review board (IRB) or research ethics committee. An IRB is an oversight body whose responsibility is to ensure the protection of human subjects involved in human subject research. They are governed by the code of federal regulations. The question of whether one must seek IRB approval for a quality-improvement activity that is not intended to contribute to generalizable knowledge is regularly debated among clinicians. As a result, it is important that clinicians not try to make the determination themselves about whether or not an activity is considered quality or human subjects research. Instead, if you have questions, engage with experts at your local institution. For more general information about research regulations and guidance related to outcomes registries, the US Department of Health and Human Services Office for Human Research Protections offers guidance regarding IRBs for research and quality improvement.6 If your organization does not have an IRB, you can either form your own research committee or contact an organization that has one to ask if you can use its services. If you are outside of the United States, seek information from your local or national regulatory agencies that govern human subjects research. When you are collecting patient data, the data collection tool must be compliant with the Health Insurance Portability and Accountability Act (HIPAA), which specifies varying levels of privacy requirements. If you are outside of the United States, familiarize yourself with the existing privacy regulations. Do your homework: Make sure that your data collection tools are secure and keep your patients’ privacy in mind. It’s your patients’ data, after all. If you plan to participate in an established registry, you should anticipate several legal steps ahead of collecting data. First, you (if you are an independent practice) or your institution will need to create a contract with the registry organization, which may be a professional group or another clinician who operates the registry. This contract outlines the terms of the engagement between you and the organization that is hosting the registry. If you work in

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

Collecting and Analyzing Outcome Data

49

a larger institution, you will likely need to find someone who has the appropriate level of authority to sign the contract. You will also need to sign a business associate agreement (BAA) with the registry organization, which is required under HIPAA in order to allow a covered entity (defined as any individual or organization that provides treatment, payment, and operations in health care) to share patient data with a noncovered entity. Finally, you will need approval from an IRB or your local research or ethics committee to participate in the registry. Regulatory and legal considerations can feel daunting and can contribute to outcome measurement inertia. Try not to let it! The key is to work with the appropriate legal, ethics, and/or regulatory team members early in the process. This part of the process can take time, so patience and tenacity are important here. Another tip is to approach legal/regulatory colleagues as partners in the process and not as barriers. This mindset will help move the process along more smoothly, as well as build partnerships for future work. Up to this point, I have focused on how to collect the data that you need for measuring outcomes. The next few sections will cover two other steps of the data management process: integrating and validating data, and extracting and analyzing.

Integrating and Validating Data Integrating data is the process of aggregating data from disparate data systems and merging those data into a single format for the end user.7 This step becomes especially important as your outcome measurement efforts scale and you are pulling data from many different sources. For the individual clinician, the integration step of data management isn’t particularly relevant because clinicians are usually the ones who record the information. I will say more about integration in the chapter on scaling outcome measurement efforts. Data validation refers to the process of verifying the accuracy and completeness of data. It is critical to ensuring that the results of your analysis are accurate over time. Unfortunately, this step is often skipped because of the time and effort that it requires, but it shouldn’t be. Complete and accurate data are critical to producing accurate analyses. To validate data means to ensure the following. 1. A data value exists where it should (i.e., there are no empty data entries). Even if the value is null, a null value should be documented; otherwise, a person analyzing the data doesn’t know whether to interpret that value as a zero or as a missing value. 2. The data are in the right format (e.g., the date is formatted consistently across entries).

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

50

How to Measure Health Outcomes

3. The data are sourced from the most accurate sources of information, and as close to the primary source of information as possible. Both Excel and REDCap allow you to create parameters around the values expected in a data entry field to minimize error upon entry. For example, if you want to record body temperature, you would specify Fahrenheit or Celsius in your data entry form or spreadsheet cell and specify a range of 90–110°F or 32–43°C. This way if someone accidently keys in numbers outside of these ranges, the database will not accept the value. Beyond this, however, spreadsheet programs like Excel have limited data validation functionality. Database programs have more because their data entry forms include prompts to users about the type of data and values expected for any given field. The data dictionary also serves as an important tool in the validation process by providing definitions for consistent data entry. The dictionary also documents the primary source of information for each variable, helping to ensure that the same source is used to retrieve a particular data point every time. As an individual clinician gathering data, take the time to routinely ensure that your data entries are accurate. If you gathered information from a medical record, ideally you would ask a colleague to review your entries. That said, this manual validation can quickly get unwieldy, which is all the more reason to familiarize yourself with database software sooner rather than later, so that you can make the switch from spreadsheets early. Analyzing your data will also reveal possible inaccuracies in data entry if the results of simple calculations don’t make sense or seem “off.” We will discuss data analysis in the next section.

Extracting and Analyzing Once you have collected and validated your data, you are ready to retrieve the information from data storage and analyze it. Your initial data analysis doesn’t need to be complicated and is really a set of basic calculations. Eventually, as your efforts grow, having a statistician on your outcomes team is a key role. To start, let’s think about analyzing the outcomes of a single patient in order to see if they have improved. The measurement timeframe will vary between months and years depending upon the medical condition and the outcome being measured. For example, for patients with diabetes whose HbA1c levels fall outside of the recommended range, you could track HbA1c values regularly while interventions are made until improvements are seen in the relatively near term (months). If improvements aren’t seen, it’s imperative that you ask “why” and adjust care as needed. For children born with a cleft lip and palate who undergo surgery as newborns, the impact of interventions (e.g., those to optimize speech development) can take longer to see (years). In cases such as this, determine if there are any intermediate outcomes that can be measured while the child is followed over a longer period of time.

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

Collecting and Analyzing Outcome Data

51

Once you have examined the outcomes of each specific patient and asked if each one is getting better, begin to analyze the outcomes of a group of your patients. Are all of your patients getting better? If not, why not? In response to this question, I have heard some clinicians say, “My patient has a terminal illness, they will not get better.” And while that might be the case, there are ways to ask if the outcomes that matter to those patients and families are being addressed. The outcomes in this case won’t include survival but could include dignity at death, having family by their side to die, minimal pain, and so on. When looking at your patients as a group, start with the basics. First, pick a timeframe, then try to answer a few basic questions about your patients from your data. 1. How many patients did you see in that time period? 2. How old were they? You will probably want to report a range here and the average age or median age. 3. What were their races and ethnicities? 4. What were their primary diagnoses? Any secondary diagnoses? Then analyze the outcome measures. If you have results from a patientreported outcome instrument, how do the patients’ scores compare to their previous scores? If you are looking at side effects for a particular medication, how many of the total patients that you saw during a particular timeframe had a particular side effect? From there you can ask, “why?” and see if you can get closer to an answer by analyzing the outcome in the context of other factors, such as age, race, or the presence of other diagnoses. That question could look something like, “Of my patients who experienced a side effect for a particular medication, how many also take other medications or fall in a certain age range?” Understanding the impact of factors such as race and age on patient outcomes is commonly known as “risk-adjustment.” Other factors often used in risk-adjustment include the severity of the patient’s medical condition, comorbidities, or psychosocial factors. I placed the term “risk-adjustment” in quotes because this is a clinical provider-oriented term, not a patient one. Patients don’t think of themselves as having risk factors, they have attributes, conditions, and want help with their health. It’s our responsibility as health care professionals to help them improve their health outcomes, while keeping their unique characteristics and/or circumstances in mind. Understanding how these factors affect outcomes is clinically important because this knowledge can inform strategies for improving outcomes. You will hear the term risk-adjustment often when measuring outcomes. When applying risk-adjustment, I encourage you to think about it as a way to help you understand why your patient outcomes aren’t as good as they could

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

52

How to Measure Health Outcomes

and should be and to further understand why a certain outcome is occurring. Risk-adjustment should not be used as a way to explain away poor outcomes by arguing, “My patients are sicker.” While some patients undoubtedly have certain circumstances that may present challenges to achieving optimal outcomes, this is not a reason to avoid trying to improve those outcomes anyway. Rather, these characteristics provide information that you need in order to better align the care that you provide to the needs of your patients. As you begin tracking individual patient outcomes, you will start and potentially stay with descriptive statistics. Descriptive statistics provide a summary of the characteristics of the cohort or segment whose outcomes you are tracking.8 They include calculations of sums, means, medians, ranges, and frequencies. These measures can help you to understand general trends and how your data relate to each other. Often clinicians who also do research apply inferential statistics to their outcome data sets. Inferential statistics are used to draw conclusions from a sample and apply those findings to either predict how a larger population will behave or make a generalized conclusion about a larger population. If you are planning to use inferential statistics and do not have training in statistics, it’s best to enlist the assistance of someone with statistical expertise. Excel can be used as an analytic tool for some basic statistical analysis, such as generating descriptive statistics, which is fine when just starting. As your outcome data set expands, you will quickly outgrow Excel’s analysis capabilities and processing power. At that point, you should turn to a statistical software program that allows users to analyze large amounts of data using statistical methods and present those data for interpretation. If using REDCap for data collection, data from REDCap can be exported for analysis using several different statistical software packages. Once you are ready to move to this phase, training in statistics and/or the help of a statistician becomes important. Without training and experience in statistical analysis, a person is more likely to make mistakes that can lead to erroneous analyses and, therefore, erroneous conclusions. Using the wrong statistical test is more common than you would think.9 The risk of this mistake is much greater with inferential statistics than descriptive. Statistical software can be used for both descriptive and inferential statistics. The most common statistical packages are SPSS, STATA, SAS, and R. Social sciences and psychology most often use SPSS; STATA is popular for economic analyses; and SAS is often used in clinical research. The least expensive of these programs is R.10 If you use Excel as your database, you will probably need to do a lot of data cleaning and preparation to format your data in order to import into one of these statistical programs for analysis. REDCap, on the other hand, has an export feature that formats data to be analyzed by all four programs, as well as an option to format data to export into Excel or a CSV (comma-separated value) file for analysis in other programs.

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

Collecting and Analyzing Outcome Data

53

As you think about selecting a statistical software program, consider the advantages and disadvantages of each, which I’ve summarized in Table 5.2, comparing SAS, SPSS, R, and STAT. These are the programs into which REDCap can export.

Getting Started As this chapter illustrates, measuring outcomes takes an investment of time and other resources. It may feel like the environment for measuring outcomes is not ideal today or in the near future, but if you are waiting for the ideal environment, enough resources, or enough time, you’ll never start. Instead, accept that conditions aren’t perfect – and then commit to measuring outcomes anyway, because you owe it to the individuals and families that you serve. You also owe it to yourself and to your clinician colleagues, whose joy at work and professionalism are enhanced by knowing how much they are helping. As your measurement efforts expand, it will be helpful to add a dedicated outcome data professional to your team. In the meantime, remember that many groups doing excellent outcome measurement work didn’t start with a large team. In the case of two urologists in Hamburg, Germany, for example, their outcome measurement journey began with one urologist tracking patients in Excel and grew over time, as I will describe in more detail in the next chapter.11 Again, I want to emphasize that you can still measure outcomes even if you don’t have many resources. As you improve your measurement and demonstrate how it translates to better care delivery and health outcomes, you may be able to devote more resources to it over time. In the next chapter, I will talk about how to deploy those resources in ways that enable you to refine and scale your outcome measurement practice.

Action Steps ▯ ▯ ▯ ▯ ▯ ▯

Choose your data collection tool. Make a list of the outcome questions that you would like to be able to answer with your data, or the insights that you hope to develop. Make a list of the variables that you plan to collect and define them, working toward building a data dictionary. Find out whom you need to talk to in your legal department and IRB/research ethics committee about the need for IRB approval for your outcome database. If you are planning to participate in a registry, make sure that you have all of the necessary contract/agreement documents needed and that you have a person to work with in the legal department of your organization. Explore statistical programs and/or support available to you. Choose the program that is best suited to answer your outcomes questions.

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

Disadvantages

• Not as easy to learn as SPSS or STATA • Annual licensing fee • Expensive • Limited functionality • Difficult to learn, which can lead to users making errors early in the analysis process

• Annual license fee • Limited to numeric or categorical data • Users cannot program new statistical functions

Advantages

• Most comprehensive in terms of statistical methods offered

• Easy to learn • User-friendly interface that resembles Excel

• Free and open source, with many free online tutorials • Can be used with any computer operating system • Users can program new statistical functions • Very active user and developer community helps provide support

• Easy to learn with free online tutorials • Wide range of statistical functions

Statistical software

SAS

SPSS

R

STATA

Table 5.2 Comparison of common statistical packages

Collecting and Analyzing Outcome Data

55

References 1. Ernest A. Codman, MD, FACS (1869–1940). American College of Surgeons. www .facs.org/about-acs/archives/pasthighlights/codmanhighlight. Accessed August 16, 2022. 2. 7 Reasons to Beware of Using Excel as a Database. Bound State Software. https://bo undstatesoftware.com/blog/7-reasons-to-beware-of-using-excel-as-a-database. Accessed August 16, 2022. 3. P. A. Harris, R. Taylor, R. Thielke et al. Research Electronic Data Capture (REDCap): A Metadata-Driven Methodology and Workflow Process for Providing Translational Research Informatics Support. J Biomed Inform. 2009; 42(2): 377–381. 4. P. A. Harris, R. Taylor, B. L. Minor et al. The REDCap Consortium: Building an International Community of Software Partners. J Biomed Inform. 2019; doi: 10.1016/ j.jbi.2019.103208. 5. V. Ehrenstein, H. Kharrazi, H. Lehmann, C. O. Taylor. Obtaining Data from Electronic Health Records. In Gliklich, R. E., Leavy, M. B., Dreyer, N. A., eds. Tools and Technologies for Registry Interoperability, Registries for Evaluating Patient Outcomes: A User’s Guide, 3rd edition, addendum 2 [Internet]. Agency for Healthcare Research and Quality, Rockville, MD, 2019; 52–79. 6. Quality Improvement Activities FAQs. HHS.gov. www.hhs.gov/ohrp/regulationsand-policy/guidance/faq/quality-improvement-activities/index.html. Accessed August 16, 2022. 7. What Is Data Integration? Definition and FAQs. OmniSci. www.omnisci.com/tech nical-glossary/data-integration. Accessed August 16, 2022. 8. Understanding Descriptive and Inferential Statistics. Laerd Statistics. https://statis tics.laerd.com/statistical-guides/descriptive-inferential-statistics.php. Accessed August 16, 2022. 9. A. M. Strasak, Q. Zaman, K. P. Pfeiffer, G. Göbel, H. Ulmer. Statistical Errors in Medical Research: A Review of Common Pitfalls. Swiss Med Wkly 2007; 137: 44–49. 10. What’s the Best Statistical Software? A Comparison of R, Python, SAS, SPSS and STATA. R-bloggers. www.r-bloggers.com/2019/07/whats-the-best-statisticalsoftware-a-comparison-of-r-python-sas-spss-and-stata/. Accessed August 16, 2022. 11. M.E. Porter, J. Deerberg-Wittram, T. W. Feeley. Martini Klinik: Prostate Cancer Care 2019. Harvard Business School Publishing, Cambridge, MA, 2019.

https://doi.org/10.1017/9781009240925.007 Published online by Cambridge University Press

Chapter

6

Scaling Outcome Measurement

Growing an Outcome Measurement Team Chapter 5 focused on how to start measuring outcomes as an individual, and primarily as an individual clinician. Eventually, that busy individual clinician won’t be able to keep up with entering all of the data needed on each patient in order to track patient outcomes and will need help maintaining the outcomes database. Tracking patient outcomes is still largely accomplished by individuals who manually abstract and record data from existing sources and/or patient surveys. Ideally, electronic health records (EHRs) would capture outcome data within the clinical workflow and interoperate seamlessly with outcome databases and/or registries. While this integration is technically possible today, reconfiguring the EHR in this way requires significant time and human and technical resources and is therefore not routinely done. So, we’ll start this chapter by discussing how to grow your outcome measurement team using a data abstraction model with one or two more people in a single clinical practice setting. We’ll end with how outcome measurement can look at scale within larger organizations that have built capabilities to leverage their EHRs for data collection and analytics. How one secures resources to grow an outcome measurement team will vary by organization and depends on decision-makers’ level of commitment to measurement and improvement. Measuring outcomes is an investment, whether of your time or of your resources to support additional outcome measurement personnel. These personnel are essential, just as you need people for reception, scheduling, and billing. A data person can be a clinical team member that enables learning and improvement for everyone on the clinical team. So, I encourage you to think of this as an essential part of pursuing excellent outcomes for the patients that you serve, rather than as an extra expense. That said, you do not have to make that entire investment up front. Rather, your outcome measurement program can start small and evolve to become more sophisticated over time. A great example of scaling measurement comes from the Martini Klinik in Germany, which provides prostate cancer care. The clinic’s outcome measurement efforts started in 1994 with a commitment to 56

https://doi.org/10.1017/9781009240925.008 Published online by Cambridge University Press

Scaling Outcome Measurement

57

measuring the outcomes of every patient treated. Initially, one urology partner collected the necessary administrative, clinical, and outcome data in an Excel spreadsheet. As Martini’s clinical practice and outcome measurement efforts grew, the hospital didn’t have positions for outcome data analysts, so the clinic hired a consultant to build its outcome database, which the clinic called “Martini Data.” With the outcomes database established, the clinic was able to create positions on their team to oversee collection and analysis of clinical and outcome data. Martini surgeons saw these colleagues as critical to their patients’ success and to the learning and improvement for each surgeon. By 2019, the outcomes team had grown to five members, including an information technology (IT) programmer, a biostatistician, and three documentation assistants with roles similar to a clinical data manager (as I describe later).1 Today, when new patients and families come to the clinic, medical providers introduce them to one of the outcomes team members and describe that person as part of the clinical team. Martini attributes its almost 100% followup survey completion rate to building a relationship between the patient and someone from the outcomes team from the beginning. When a patient receives a follow-up phone call from a documentation assistant, the patient knows that person by name. The call feels meaningful, as opposed to mechanized or devoid of caring, like many postcare experience surveys.

Scaling as a Single Practice Because most outcome tracking still requires manual data abstraction and recording, initial scaling efforts typically involve adding either a clinical data manager or a computer programmer (or both, if resources allow) to your practice. If you are at the point that you’re ready to scale up your outcome measurement program, it is important to recruit someone who has or can easily learn database skills. For this reason, many people hire a database programmer. Database programmers typically have nonclinical backgrounds and strong technical skills. Having a programming expert can be especially helpful if you participate in a national registry, as registries typically use outside software platforms that are either provided through a vendor or built by a local institution; having someone who can easily communicate with technical staff on the registry side is thus helpful. Programmers are also able to run audit reports and data validation routines that can help ensure data integrity and data quality. They can often write code to automate data-quality checks or data-validation routines. With tools such as REDCap available, it may not be necessary to hire someone with technical expertise to build your outcome database. In that case, a clinical data manager may be more appropriate. Clinical data managers typically have a clinical background, ideally in the same clinical field as the practice that they work in, as well as good analytical and critical thinking skills

https://doi.org/10.1017/9781009240925.008 Published online by Cambridge University Press

58

How to Measure Health Outcomes

and attention to detail. Their clinical background gives them better context for understanding which data elements and outcome measures need to be tracked. While they generally do not have the same level of technical knowledge as a database programmer, they can often learn how to maintain the outcome database. Part of their role, at least at the beginning of scaling efforts, includes abstracting data from health records, making follow-up calls to track patients longitudinally, and maintaining the data dictionary. The clinical data manager also usually takes on an informal leadership role in managing the outcomes program. If you have a large practice, the clinical data manager will likely be a full-time role. If you are in a smaller practice, having someone in this role part-time may be adequate, at least initially. Avoid asking clinical team members whose primary responsibility is patient care also to be data collectors. Although it seems logical for a person already seeing patients to collect data at the same time, abstracting data for outcome tracking takes a particular mindset. First, the person must be willing to sit in front of a computer most of the day, which clinicians who enjoy patient interaction often do not want to do – especially as they already spend so much time in front of the computer charting. Second, the data collector needs to enjoy being a sleuth of sorts. Ensuring that the data recorded reflect the reality of clinical care requires reading multiple notes from different sources, verifying consistency between the notes, and identifying any aberrations. A person whose primary job is to take care of patients won’t have time to do this. Data collection responsibilities can’t simply be added on to another full-time job. There are a few exceptions to this rule. First, some clinical data are so nuanced that it may be best for clinicians to record those particular data elements themselves. For example, when measuring congenital heart surgery outcomes, the surgeon is really the ideal person to complete the diagnosis and procedure field in the outcomes database after surgery because they observed the congenital defect and know exactly which operation they performed. Second, there are times when the data are so simple and straightforward that it’s relatively easy to incorporate their collection into the clinical workflow. For example, I have seen clinicians use standardized paper data collection forms that include data definitions as a way to have every member of the clinical team participate in data collection. However, this practice only works if everyone agrees to participate. And it’s important to remember that data collection is only the first step. Someone must still enter the data in a database, ensure data collection is complete, conduct data analysis, and perform dataquality assurance. Third, if you can extract data from the EHR, this eliminates the need for manual data collection from the record. In that case, the clinical teams are already participating in data collection through their routine documentation responsibilities. In most cases today, however, even if outcome data collection

https://doi.org/10.1017/9781009240925.008 Published online by Cambridge University Press

Scaling Outcome Measurement

59

forms are built into the EHR, clinicians must go outside the typical documentation workflow to complete them. As a result, outcome data may not be recorded consistently. If your practice is connected to a larger health system, you may be able to automate the collection of demographic information from the patient registration system. This can significantly reduce the amount of manual data entry required. In general, the fewer the people involved in collecting data, the better. When multiple people complete data collection forms, data fidelity and data reliability can be at risk. Yes, quality-assurance checks can and should be done after data collection, but it is much more efficient – and better for data quality overall – to collect it as accurately as possible up front, and this means limiting the number of people who handle the data. The addition of either a clinical data manager or database developer to your outcome measurement efforts constitutes the beginning of your core outcomes team – even if this is just two of you! This core team should meet regularly with the broader clinical care team at least once monthly to review outcomes and discuss improvement opportunities. This clinical care team should involve everyone on the team, including front- and back-office staff, because everyone has a role to play in achieving Capability, Comfort, and Calm for the people seeking care from your practice. As your data measurement efforts grow, the team should also be involved in selecting new outcome measures.

Scaling across an Organization The next level of scale is to create an “outcomes service” that supports outcome measurement for an entire department or organization. The service comprises a combination of clinical data managers, clinical data or outcome specialists (I will describe this role in more detail later), a statistician, and additional IT resources and expertise, such as a system analyst (also described in more detail later). This team can support outcomes measurement across multiple conditions in partnership with a clinical champion for each condition. I will describe how an outcomes service supports measurement at scale in terms of the essential data management functions it performs, not based on what it should look like, because how an organization chooses to staff and organize this type of service will vary. In other words, I’m writing from the perspective of what is needed to get the work done. Your organization’s leaders will decide how the necessary team members come together to do that work. I will provide some considerations related to governance toward the end of this chapter. For now, let’s focus on the key functions that the outcomes service needs to perform in order to enable outcome measurement, as well as how it can leverage the EHR and data analytic platforms to support these functions. Figure 5.1 serves as a helpful point of reference here because

https://doi.org/10.1017/9781009240925.008 Published online by Cambridge University Press

60

How to Measure Health Outcomes

the basic components of the data management process are the same whether you are one person manually collecting data for outcome measurement or a team using an EHR and a data analytic system. In fact, as your team grows and more and different technologies become available, understanding basic principles of sound data management in order to enable valid and reliable outcome measurement becomes even more critical. The technology that you use should be especially scrutinized in order to ensure that you are tracking the outcomes of each individual person and that the data are valid, reliable, and reproducible.

Collect, Record, and Store Data Scaling outcomes measurement at the departmental or organizational level often involves a heavier reliance on the EHR. As we’ve previously noted, EHRs are essentially electronic versions of paper medical records and were designed first and foremost to streamline billing, not to inform clinical care improvements. That said, with the help of IT resources, there are ways that the EHR can become a somewhat more helpful tool for data collection. With an EHR, data can be collected through clinical documentation forms, patient portals, iPads or other tablet-type devices, kiosks in waiting rooms, or mobile apps. If you plan to collect data at multiple points of the patient encounter, make sure that your analytics team knows this when they try to access the data for reporting so that they can integrate data from all the points of data entry. If data needed for outcome measurement aren’t captured through clinical documentation, you will need a person to help manually collect the missing data elements. Some larger EHR systems maintain a central library of patient-reported outcome (PRO) collection forms that can be added into documentation workflows upon request at the local level. Another option here is to build custom outcome data collection forms into the EHR system.2 To take this approach, you will need help from someone in IT who is authorized to make changes to the EHR and has the technical skills to build the forms. These experts are often called system analysts.

Integrate and Validate Data Even though data is recorded into the EHR, extracting that information for reporting or analytics requires additional data technology and IT expertise. Most EHR systems’ reporting capabilities are limited because the reports tend to be static and difficult to change. To harness the full power of the data stored in the EHR, you need some type of data warehouse capability that integrates data from different parts of the medical record, as well as from other data systems within the organization. You also need IT experts, often called data architects, who can write code to extract the data from the data warehouse.

https://doi.org/10.1017/9781009240925.008 Published online by Cambridge University Press

Scaling Outcome Measurement

61

When integrating data manually, you abstract data from different parts of the medical record and enter the data into the outcomes database. Data warehouse technology essentially uses computer code to automate this process. While writing this code falls to the IT team, your scaled-up outcomes service should also include a clinical person who can explain to the data architects which data need to be integrated and why. This role is typically called either a clinical data specialist or clinical outcome specialist. This person should be someone with a clinical background and either data analytics or informatics experience who can translate between the technical language of the data architects and the clinical language of the care providers who are leading the outcome measurement effort. The data architects and clinical data specialists work closely together almost daily to try to make sure that the clinical data coming out of the EHR system for outcome analysis is accurate. Sometimes this clinical role can be filled by the clinical data manager described earlier in the chapter, depending on that individual’s level of data analytics or informatics expertise. Typically, clinical data specialists or clinical outcome specialists have master’s degrees, while clinical data managers have bachelor’s degrees. This is a good place to pause and explain the role of the clinical champion. The clinical champion usually is not a full-time member of the outcomes service; rather, they belong to a clinical department that wants to measure outcomes and may have a formal role in leading outcome measurement efforts for that department. The clinical champion is typically a physician, although nurses or other clinicians may fill this role as well. Either way, the clinical champion plays a critical role in partnering with the clinical data specialist and IT experts in order to design new data collection tools. The clinical champion also serves as a workflow expert when changes to EHR data entry procedures are needed to capture outcome data and helps ensure that data integrated in the EHR system are accurate for outcomes analysis upon extraction. This partnership between the core outcomes team, IT experts, and clinical champion can be thought of as an outcome measurement triad (Figure 6.1). (As depicted with the dotted lines in this figure, IT resources are typically partners to as opposed to staff on the core outcomes team.) An important note about clinician readiness to measure outcomes: In my experience, most health care providers really want to measure outcomes – they just don’t have the support and resources to do so. This is where administrative and clinical leaders can particularly accelerate outcome improvement by making it an organizational priority and resourcing it. Administrative and clinical leaders can help by being engaged in measurement; asking for regular updates; and supporting clinicians with time to do outcome improvement work, as well as with human resources and technical infrastructure. If tracking outcomes becomes an organizational priority, the organization can often refocus existing resources on outcome measurement and improvement.

https://doi.org/10.1017/9781009240925.008 Published online by Cambridge University Press

62

How to Measure Health Outcomes

CORE OUTCOMES TEAM CLINICAL DATA MANAGERS CLINICAL DATA SPECIALISTS

IT PARTNERS DATABASE PROGRAMMERS SYSTEM ANALYSTS DATA ARCHITECTS

CLINICAL CHAMPION FOR A PARTICULAR CONDITION

Figure 6.1 The outcome measurement triad – a clinical data specialist from the core outcomes team, IT partners, and a clinical champion. The three members of the triad are typically from different departments who come together to measure outcomes.

Extract and Analyze Data When using legacy EHR and data analytic systems, one of the most challenging and time-consuming aspects of outcome measurement is extracting an accurate list of patients within your segment of interest. With manual abstraction, creating an accurate patient list is not as problematic because the act of manually logging each patient record means a person with some knowledge about the goals of outcome measurement – whether it’s a clinician or a member of the outcomes team – is making a logic-based judgment about which patients to include. When using electronic sources, however, you must provide the “logic” ahead of time – that is, the specific parameters that programmers or data architects will use to find the correct patient records within the data system. For example, if you are looking for a list of patients who have undergone a particular procedure, you can often search your electronic data system for either a CPT (current procedural terminology) or ICD (International Classification of Diseases) procedure code, performed within a particular time frame. This search and logic are relatively straightforward.1 Even with relatively straightforward procedural cases, though, you may still want more nuance in defining your patient segment than just that a particular procedure was performed. For example, defining the segment by whether or not a person had an appendectomy doesn’t enable one 1

CPT codes are preferable for this type of search because they are documented by clinicians and relatively reliable for identifying that a specific procedure was carried out. ICD-10 also has procedure codes, but these are coded by billing specialists and thus have a greater chance of error. Either way, however, it is important to ensure that you have access to current lists of CPT and ICD codes. They are available in regularly updated code books and online.

https://doi.org/10.1017/9781009240925.008 Published online by Cambridge University Press

Scaling Outcome Measurement

63

to ask the question, “How many people presented with abdominal pain that was correctly diagnosed as appendicitis?” It is more difficult to identify a segment using diagnostic codes; often, the field is not reliably captured with enough specificity in the medical record, and while the EHR “problem list” may contain diagnostic information, the ways in which problems are documented in that section can vary tremendously. Nevertheless, putting in time upfront to determine whether you can extract diagnostic information can yield greater benefits in long-term analysis. For patients who don’t undergo procedures, creating a list of inclusion criteria in order to identify patients in a particular segment can take some extra thought and creativity. For example, if you are interested in looking at outcomes for children presenting with acute asthma exacerbation in the emergency room, simply using a diagnosis of asthma could potentially return a list of any patient who at any time received a diagnosis of asthma. You would need to add a few more search parameters in order to narrow your search. In this case, searching for children with 1. a diagnosis of asthma; 2. a record of receiving certain medications used to treat an asthma exacerbation (terbutaline, corticosteroids, etc.); and 3. a location of care in the emergency department will return a more specific list. Count on refining your search parameters several times – this is a highly iterative process, requiring input from all members of the outcomes triad. Details about how you defined the patient segment should be documented in your data dictionary. Further complicating the creation of an accurate patient list is the fact that numerous versions of patients’ records can exist within one system. For example, different records can be generated depending on site of care (emergency room, outpatient, or inpatient) if the EHR system uses different modules or systems for each. Multiple billing records are also often generated – one for the hospital, for example, and another by the clinical provider. Unfortunately, it is not uncommon for one health system to use multiple EHR systems that do not interoperate. In this setting, it is often more efficient to create an outcome database outside the EHR environment, prospectively identify patients, and manually track outcome data. I have mentioned a few times above the need to have help from IT staff in order to leverage the capabilities of the EHR for data collection and analytics. More specifically, you will need access to the two types of IT experts mentioned previously: someone who can help you get outcomes tracking data into the EHR (system analyst) and someone who can help you get it out (programmer or data architect). The IT experts who perform these functions are highly trained and skilled in information systems, computer science, and/or programming. I’m abridging their roles here with respect to outcome measurement in order to keep things simple. My phrasing around needing “access to two types of IT experts” is deliberate. In all likelihood, these staff resources will be shared across an organization. It is

https://doi.org/10.1017/9781009240925.008 Published online by Cambridge University Press

64

How to Measure Health Outcomes

unusual (although not impossible) for a single clinical team to have its own dedicated IT resources. Some organizations have “super-users” within their clinical teams that have limited access to the EHR in order to build custom reports and certain data collection forms. These super-users are typically clinicians who received extra training from the EHR vendor in order to be able to perform these functions. Other organizations, however, have a more centralized approach to how IT resources are managed. This can become a point of inertia for outcome measurement because requests for creating data collection forms or making other enhancements to the EHR are taken on a first-come, first-served basis or prioritized against other requests, which can create a long lead time. The same is true for getting data requests completed. For this reason, some organizations have invested resources to create self-service reporting tools that allow end users to manipulate and visualize data on their own through software applications like Tableau. These tools also can generate descriptive statistics and can usually export to comma-separated value (CSV) or Excel files that can be uploaded into statistical programs for further analysis. Some visualization tools have plug-ins for statistical packages like R. One last note on EHRs: Not every EHR and data analytic system has the capabilities to do what I have just described. It’s important to get to know your own organization’s IT team and learn what is possible.

Share Data and Improve The final function of the data management process is to share data and improve. Organizations that have implemented outcome measurement at scale will likely have formal clinical outcome improvement teams. These teams are formed as needed and may come together for different reasons. Perhaps an individual clinician wants to start measuring outcomes for a particular condition and has a leadership mandate to do so. Or perhaps opportunities for improvement are identified through regulatory reviews or internal analysis of organizational data that leads to new outcomes measurement teams forming. For example, at my previous organization, when a key process analysis using data from the data warehouse showed high levels of care variation for children with appendicitis and associated high costs, a care process team was formed to address the problem, improve outcomes, and thereby bring costs down. This care process team was supported by the outcomes service. Finally, I want to share a few thoughts on what to expect when you present outcome data to a broader audience, such as other physicians in the relevant clinical department. Although the purpose of measuring outcome data is improvement, not criticism, seeing performance data may initially feel punitive or threatening to those who haven’t had much previous experience with measurement. It’s important to be prepared for people’s potential reactions when presented with outcome data for the first time. Don Berwick, the former

https://doi.org/10.1017/9781009240925.008 Published online by Cambridge University Press

Scaling Outcome Measurement

65

president of the Institute for Healthcare Improvement, described these reactions as the “Stages of Data Acceptance.”3 They are listed here. Stage one: “The data are wrong.” In this stage, people don’t trust the data presented to them. They look for flaws in how it was collected, if the risk adjustment was correct or done at all, and make any number of critiques of how the data were gathered and analyzed. This is why it is critical that before presenting data, you and/or your team have done the necessary work in data collection and validation in order to ensure the data are accurate. If one person in the room can find a hole in the data, it calls into question the credibility of the entire effort. Once credibility is lost, it is difficult to regain. Stage two: “The data are right, but it’s not a problem.” In this stage, people accept the data presented to them but pose myriad excuses for why the outcomes are not as good as they could be. This is often the stage where people say, “Our patients are sicker than other patients.” Stage three: “The data are right, it’s a problem, but it’s not my problem.” In this stage, people have accepted the data and realized that improvement is needed but have decided that it is someone else’s responsibility. Someone or something else is to blame. Stage four: “The data are right, it’s a problem, and it’s my problem.” In this stage, people accept the data presented, believe improvement is needed, and understand that it is their responsibility to find solutions. Potential resistance to data is one of the realities of outcome measurement work. My advice is to keep pushing ahead with the willing. Continue to convene the team and share the data. An important aspect of all this work is to keep showing up. Eventually, the least supportive may become the champions – I have seen it happen in my work.

Governance and Leadership At the outset, it is important to stand up an effective governance and organizational structure for the outcomes service. Where in the organization does this service fit? A natural home for an outcomes service is within the quality department, but there are other options depending on how your organization is structured, such as a department focused on health care transformation or valuebased care. Regardless of where it is housed, the outcomes service should report to someone at the executive level, such as the chief quality officer, chief transformation officer, or someone with a similar title. The service may also have a dottedline reporting relationship to the chief medical officer or another physician executive. Other key considerations in establishing outcomes measurement and

https://doi.org/10.1017/9781009240925.008 Published online by Cambridge University Press

66

How to Measure Health Outcomes

improvement at scale within an organization are to ensure that the team has: 1. access to resources; 2. a clear partnership with IT; 3. visibility from senior leadership; and 4. clearly delineated partnerships with operational teams. It is also important to recognize that informal influence can be just as, if not more, critical to successfully implementing outcomes measurement across the organization than formal governance structures. Spend time building your credibility in relationships, and thereby your influence. It is time well spent.

References 1. M. E. Porter, J. Deerberg-Wittram, T. W. Feeley. Martini Klinik: Prostate Cancer Care 2019. Harvard Business School Publishing, Cambridge, MA, 2019. 2. K. Carberry, Z. Landman, M. Xie et al. Incorporating Longitudinal Pediatric Patient-Centered Outcome Measurement into the Clinical Workflow Using a Commercial Electronic Health Record: A Step toward Increasing Value for the Patient. J Am Med Inform Assoc 2016; 23: 88–93. 3. Improvement Tip: Take the Journey to “Jiseki.” IHI – Institute for Healthcare Improvement. www.ihi.org:80/resources/Pages/ImprovementStories/Improvement TipTaketheJourneytoJiseki.aspx. Accessed August 17, 2022.

https://doi.org/10.1017/9781009240925.008 Published online by Cambridge University Press

Chapter

7

Now the Journey Begins

Now that you have finished reading this guide about how to measure outcomes, I want to conclude with a reminder that measuring outcomes is not an end in and of itself. It is a launching point – the beginning of a conversation with a patient, another provider, or yourself. Putting that information to work for individuals and families requires bold action, good judgment, and often courage. The courage to face numbers that you don’t like seeing or didn’t think you’d see. The courage to make the changes necessary in order to help the people seeking your help. The courage to be the voice for patients and families when they can’t be in the room when decisions about them are made. The numbers we collect and analyze as we measure outcomes represent people – real people with real lives that will go on after we meet them – and hopefully we have helped them along their way.

67

https://doi.org/10.1017/9781009240925.009 Published online by Cambridge University Press

Index

Please note: page numbers in bold type indicate figures or tables. administrative data, sources of, 21 anomalous aortic origin of the coronary artery (AAOCA), children with as segment of a cohort, 18 asthma, measuring management of, 8 baseline data, 12, 17, 21 benefits of gathering, 17 collection of, 21 common endpoints to examine, 21 historical/retrospective data, 24–25 requesting and analyzing, 24–25 timeframe example, 24 prospective data, 25–26 timeframe example, 25 sources of administrative and clinical data, 21 types of data and where to find it, 22–23 basic vocabulary for outcome measurement, 17 benefits of outcome measurement appendicitis example, 63, 64 asthma example, 8, 63 Cincinnati Children’s Hospital Medical Center, tracking progress in OCD treatment, 9, 25, 26, 34 diabetes and pregnancy example, 31 electrophysiologist practice example, 13 for informed patient choice of treatment options, 14, 15 migraine care example, 20 statin prescribing example, 14

calm, concept of, 8 Capability, Comfort, and Calm outcome framework, 30, 33 baseline information and improvements in care, 12 developers, 3, 7 the framework, 7–8 size of measure set, 33 taxonomy, 29 using, 27 capability, examples of, 8 Centers for Medicare and Medicaid Services (CMS), Star ratings, 9 centers of excellence based on outcomes, 13 children’s heart surgery, outcomes team, end goal, 5 Cincinnati Children’s Hospital Medical Center, tracking progress in OCD treatment, 9, 25, 26, 34 cleft lip and palate, results of surgical repair as example of outcomes that matter most, 26 Cleveland Clinic, 5 clinical champion, role in the outcome measurement team, 61 clinical data, sources of, 21 clinical data managers, role in scaling outcome measurement, 57–58 clinical registries for data collection and analysis, 46–47 as source of potential measures, 32 clinician leadership, and identification of outcome measures, 29 Codman, Ernest, 1, 38 cohorts, concept of, 18

68

https://doi.org/10.1017/9781009240925.010 Published online by Cambridge University Press

Index cohort/segment identification, 18 for administrators, 19–20 in payer organizations, 19–20 for clinicians, 19 migraine care example, 20 see also patient segments comfort, concept of, 8 congenital heart disease anomalous aortic origin of the coronary artery (AAOCA), 18 children with as cohort of patients, 18 critical heart disease, natural history of the disease, 7 critical information provision, for patients and families, 14–15 improvements in outcomes for children with, 7 congenital heart surgery data collection, 40 minimal data set, 33 typical questions about, 19 current landscape of measurement in health care, 1–3 current performance, understanding of as reason to measure outcomes, 12–13 data, baseline, see baseline data data collection and analysis, 36, 37 action steps, 53 additional data, examples of, 33 best person for the job, 58–59 clinical registries, 46–47 Codman’s method, 38 collection and storage, 37–39 congenital heart surgery example, 40 data management process, 37, 62 electronic health records, see electronic health records (EHRs) ethical and legal issues, 47–49 extraction and analysis, 50–53 getting started, 53 integration and validation, 49–50, 60–61 IT staff requirements, 63

69

keeping it simple, 36 manual extraction vs electronic, 62–64 measurement timeline, 47 methods, 37–39 privacy regulations and, 48 scaling, 60 self-service reporting tools, 64 software choosing, 38, 44–46 comparison, 54 Excel, see Excel spreadsheets REDCap, see REDCap database validation tools, 50 “Stages of Data Acceptance” (Berwick), 65 unit of analysis, 40–41 data dictionary concept of, 40 examples, 46, 63 function, 40, 50 maintaining, 58 REDCap, 43 data integrity aspects of, 38 concept of, 37 database architects, potential role in validation of data, 61 database programmers, potential role in scaling outcome measurement, 57 descriptive statistics, 52 developing outcome measures, “Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer,” 32 Die Kaufmännische (KKH) (German health plan), 20 Donabedian, Avedis, 2 Donabedian framework, 2 electronic health records (EHRs) building custom outcome data collection forms into, 60 extraction of data from, 58, 59–60, 63

https://doi.org/10.1017/9781009240925.010 Published online by Cambridge University Press

70

Index

benefits, 58 comparison with manual abstraction, 62–64 IT staff requirements for data collection and analysis, 63 limitations, 60, 64 producers of, 32 role of the clinical champion, 61 “super-users,” 64 employer, benefits of outcome measurement, Walmart example, 13 ER visits, examples of outcomes that matter most, 31 ethical and legal considerations in data collection and analysis, 47–49 ethical obligations to measure outcomes, 11 Excel spreadsheets, 38–39 comparison with REDCap database, 38, 41–43, 44–46, 50, 52 examples, 36, 42 Martini Klinik’s use of, 57 using a data dictionary with, 40 existing measures, adapting, 34 Experience Group™ sessions, 26 fertility treatment eligibility, as example of outcomes that matter most, 31 gender, collection of data on, 39–40 governance and leadership, effective, 65–66 health, WHO definition, 3, 8 health care definition and purpose, 1, 3 economic considerations, 12 meaning of “quality” in, 1 outputs of, 2 health care administrators in payer organizations, 19–20 role in cohort/segment identification, 19–20

health care measurement current landscape, 1–3 reorienting to patient health measurement, 3 health screenings, asthma, 8 HealthMeasures (Person-Centered Assessment Resource), 32 historical/retrospective data choosing a time period, 24 requesting and analyzing, 24–25 timeframe example, 24 hospice care, 7 identifying outcome measures, 29–32 action steps, 34–35 adapting an existing measure, 34 clinician leadership and, 29 narrowing the measure set, 33–34 narrowing the measure set, Modified Delphi method, 34 refining your list, 32–33 incentives, as reason for measuring outcomes, 15 inferential statistics, 52 Institute for Healthcare Improvement, 65 institutional review boards (IRBs), 48 integration and validation of data, 49–50 scaling, 60–61 interdisciplinary care teams, cohort/ segment identification and, 20 International Consortium for Health Outcomes Measurement (ICHOM), 31 Knee Injury and Osteoarthritis Outcome Score (KOOS), 29 adaptation of survey questions, 34 leadership, effective governance, 65–66 learning and improvement opportunities, identifying, 13 legal and ethical considerations in data collection and analysis, 47–49

https://doi.org/10.1017/9781009240925.010 Published online by Cambridge University Press

Index Martini Klinik, Hamburg, outcome measurement process, 13 team, 56–57 Microsoft Access, 38, 45 Microsoft Excel, see Excel spreadsheets migraine care, creation of integrated practice unit, 20

71

external incentives for measuring, 15 fertility treatment eligibility example, 31 identifying, 8, 17, 26–27 on-time surgery starts example, 31 surgical repair of cleft lip and palate example, 26 terminal patients, 51 outputs of health care, 2

Nightingale, Florence, 1 obsessive-compulsive disorder (OCD), tracking progress in treatment, 9, 25, 26, 34 on-time surgery starts, as example of outcomes that matter most, 31 organization level scaling, see also outcome services, 59–60 outcome, definition, 17 outcome measurement as a launching point, 67 basic vocabulary for, 17 care cycle, 9–10 creating a foundation for, 3 end goal, role in health care and, 4–6 incentives, 15 individual patient level, 9 outcome measurement triad, 62 outcome measures definition, 17 developing new measures, 32 Donabedian framework, 2 online source of, 32 reasonable size of measure set, 33 sources of, 31–32 outcome tracking, vs outcome research, 36 outcomes of health care, defining, 7 outcomes service components, 59 key functions, 59 personnel requirements, 61 position in the organization, 65 outcomes that matter most categories, 8, 17, 29 ER visits example, 31 Experience Group™ sessions, 26

patient choice, benefits of outcome measurement for informing, 14, 15 patient cohorts, concept of, 18 patient health measurement, reorienting from health care measurement, 3 patient list, challenges of creating accurate list from available data, 62–63 patient satisfaction surveys, function, 9 patient segments examples, 18, 18 identifying electronically vs manually, 63 see also cohort/segment identification patient-centred outcome measurement, increasing acceptance, 2 patient-reported outcomes (PROs) collection forms, 60 growth in use of, 1 surveys, 21, 29 Patient-Reported Outcomes Measurement Information System (PROMIS), 32 patients and families, providing critical information to, 14–15 payer organization, role in outcome measurement, 19–20 privacy considerations in data collection, 48 process measures, Donabedian framework, 2 prostate cancer care benefits of measuring outcomes, 13 scaling measurement example, 56

https://doi.org/10.1017/9781009240925.010 Published online by Cambridge University Press

72

Index

race, collection of data on, 11, 39–40, 51 reasons to measure outcomes, 11 economic considerations, 12 ethical obligations, 11 learning and improvement opportunities, identifying, 13 patients and families, providing critical information to, 14–15 resource allocation, 12–13 understanding current performance, 12–13 REDCap database, 43, 45 built in data, 32 comparison with Excel spreadsheets, 38, 41–43, 44–46, 50, 52 data dictionary, 43 examples, 44, 45, 46 technical skills, 57 reorienting to patient health measurement, 3 reporting tools, self-service, 64 research ethics committee, 48 resource allocation, as reason for measuring outcomes, 12–13 risk-adjustment, 51, 52 scaling outcome measurement collection and storage, 60 effective governance and leadership, 65–66 extraction and analysis, 62–64 integration and validation, 60–61 organization level scaling, 59–60 see also outcome services prostate cancer care example, 56 sharing and improvement, 64–65 single practice level scaling, 57–59 team development, 56 segments, patient, see patient segments self-service reporting tools, 64 sex, collection of data on, 39–40 single practice level scaling, 57–59 software choosing, 38, 44–46 Excel, see Excel spreadsheets

platforms used by national registries, 57 REDCap, see REDCap database statistical packages, 52–53 comparison, 54 Tableau, 64 validation tools, 50 “Stages of Data Acceptance” (Berwick), 65 starting points, 17 action steps, 27 baseline data gathering, see baseline data cohort/segment identification, see cohort/segment identification outcomes that matter most, see outcomes that matter most statistics, descriptive and inferential, 52 structural measures, Donabedian framework, 2 surgical repair of cleft lip and palate, results of as example of outcomes that matter most, 26 team development, 56 terminal patients, outcomes that matter most to, 51 treatment options, benefits of outcome measurement for informed patient choice, 14, 15 United States (US) CMS Star ratings, 9 Department of Health and Human Services Office for Human Research Protections, 48 Health Insurance Portability and Accountability Act (HIPAA), 48 validation and integration of data, 49–50 scaling, 60–61 validation tools, 50 in vitro fertilization (IVF) clinics, and misrepresentation of clinical outcomes, 15

https://doi.org/10.1017/9781009240925.010 Published online by Cambridge University Press