Mastering the Instructional Design Process: A Systematic Approach [5 ed.] 9781118947135

Table of contents :
Contents
Preface to the Fifth Edition
Acknowledgments
About the International Board of Standards for Training, Performance, and Instruction
Advance Organizer
About This Book
PART ONE: OVERVIEW
1 An Overview of Instructional Design
PART TWO: PLANNING AND ANALYZING FOR INSTRUCTIONAL DESIGN
2 Conducting a Needs Assessment
3 Collecting and Analyzing Data for Instructional Design Projects
4 Identifying the Target Population and Environmental Characteristics
5 Using Analytical Techniques to Determine Instructional Content
6 Analyzing the Potential Use of Existing and Emerging Technologies
PART THREE: DESIGNING AND DEVELOPING FOR INSTRUCTIONAL DESIGN
7 Using an Instructional Design Process Appropriate for a Project
8 Organizing Instructional Programs and/or Products
9 Designing Instructional Interventions
10 Planning Noninstructional Interventions
11 Selecting or Modifying Existing Instructional Materials
12 Developing Instructional Materials
13 Designing Learning Assessments
PART FOUR: EVALUATING AND IMPLEMENTING INSTRUCTIONAL DESIGN
14 Evaluating Instructional and Noninstructional Interventions
15 Revising Instructional and Noninstructional Solutions Based on Data
16 Implementing Instructional and Noninstructional Interventions
PART FIVE: MANAGING INSTRUCTIONAL DESIGN
17 Applying Business Skills to Managing the Instructional Design Function
18 Managing Partnerships and Collaborative Relationships
19 Planning and Managing Instructional Design Projects
PART SIX: PREDICTING THE FUTURE OF INSTRUCTIONAL DESIGN
20 Predicting the Future of Learning and Instructional Design
About the Authors
References
Index

Citation preview

Mastering the Instructional Design Process

Fifth Edition

Mastering the Instructional Design Process A Systematic Approach A REVISED EDITION BASED ON AN ADAPTATION OF INSTRUCTIONAL DESIGN COMPETENCIES: THE STANDARDS (4TH ED.)

William J. Rothwell G.M. (Bud) Benscoter Marsha King Stephen B. King

This book is printed on acid-free paper. ∞ Copyright © 2016 by Rothwell, Benscoter, King, & King. All rights reserved The material on pages 4, 19, 20, 61, 77, 90, 119, 145, 171, 189, 196, 216, 228, 234, 263, 271, 272, 291, 317, 318, 334, 335 are printed with permissions from IAP – Information Age Publishing, Inc. All rights are reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey Published simultaneously in Canada No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at www.wiley.com/go/permissions. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor the author shall be liable for damages arising herefrom. For general information about our other products and services, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002. Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com. Library of Congress Cataloging-in-Publication Data Names: Rothwell, William J., 1951–author. Title: Mastering the instructional design process : a systematic approach / William J. Rothwell, G.M. (Bud) Benscoter, Marsha King, Stephen B. King. Description: Fifth edition. | Hoboken, New Jersey : John Wiley & Sons, Inc., [2016] | Includes bibliographical references and index. Identifiers: LCCN 2015036837 | ISBN 9781118947135 (cloth); ISBN 9781118947159 (ePDF); ISBN 9781118947142 (ePub) Subjects: LCSH: Employees—Training of. | Instructional systems—Design. Classification: LCC HF5549.5.T7 R659 2016 | DDC 658.3/12404—dc23 LC record available at http://lccn.loc.gov/2015036837 Cover Design: Wiley Cover Image: © iStock.com/aleksandarvelasevic Printed in the United States of America 10 9 8 7 6 5 4 3 2 1

William J. Rothwell: I dedicate this book to my wife, Marcelina V. Rothwell. She is the wind beneath my wings.

Bud Benscoter: I dedicate this book to the inspiring and inspired professors, colleagues, and students who influenced my commitment to the field of instructional systems design.

Marsha and Steve King: We dedicate this book to our son, John, who inspires us every day.

CONTENTS

Preface to the Fifth Edition xiii Acknowledgments

xix

About the International Board of Standards for Training, Performance, and Instruction xxi Advance Organizer About This Book

PART ONE: OVERVIEW

xxiii

xxvii

1

1 An Overview of Instructional Design 3 Instructional Design: An Emerging Profession 4 Instructional Design: Focused on Establishing and Maintaining Efficient and Effective Human Performance 5 Instructional Design: Guided by a Model of Human Performance 7 Instructional Design: Carried Out Systematically

11

Instructional Design: Based on Open Systems Theory 11

vii

viii

Contents

Instructional Design: Oriented to Finding and Applying the Most Cost-Effective Solutions to Human Performance Problems 13 Criticisms of Traditional Instructional Approaches 14

PART TWO: PLANNING AND ANALYZING FOR INSTRUCTIONAL DESIGN 17 2 Conducting a Needs Assessment

19

Defining Terms 23 Steps in Developing Needs Assessment 29 Developing a Needs Assessment Plan: A Case Study 35 Solving Problems in Conducting Needs Assessments 36 Identifying Instructional Problems 38 Ethical Issues in Needs Assessment 38

3 Collecting and Analyzing Data for Instructional Design Projects

41

The Nature of Data 41 The Data Collection Process 45 Data Analysis 58

4 Identifying the Target Population and Environmental Characteristics 61 Selecting Learner Characteristics for Assessments 62 Selecting Learner Characteristics: A Case Study Example 71 When Should Learner Characteristics Be Assessed? 73

5 Using Analytical Techniques to Determine Instructional Content Identify Content Aligned with Needs Assessment Findings 91 Elicit, Synthesize, and Validate Content from Subject Matter Experts

91

Analyze Instructional Products to Determine the Adequacy of Content, Instructions, and Learning to Determine Content Coverage 92 Determine Subordinate and Prerequisite Skills and Knowledge 93 Analyze Content

97

6 Analyzing the Potential Use of Existing and Emerging Technologies 98 E-Learning

100

The Effectiveness of Today’s Instructional Technology 110

90

Contents

ix

PART THREE: DESIGNING AND DEVELOPING FOR INSTRUCTIONAL DESIGN 117 7 Using an Instructional Design Process Appropriate for a Project

119

Selecting or Creating an Instructional Design Process Based on the Project 123 Modifying the Instructional Design Process as Project Parameters Change 123 Describing a Rationale for the Selected, Created, or Modified Instructional Design Process 124

8 Organizing Instructional Programs and/or Products 125 Determine the Overall Scope of Instructional Programs 125 Specify and Sequence the Anticipated Learning and Performance Outcomes 138

9 Designing Instructional Interventions

145

Identifying Instructional Strategies Aligned with Instructional Goals and Anticipated Learning Outcomes 149 Choosing an Appropriate Instructional Strategy 155 Choosing Media and Delivery Methods 159 Selecting Delivery Modes 162 Appreciating the Learner’s Perspective: A Brief Overview of Cognitive Strategies 164 Using Appropriate Message, Visual, and Motivational Design Principles Accommodating Social, Cultural, Political, and Other Individual Factors Influencing Learning 169

10 Planning Noninstructional Interventions

171

Identifying Appropriate Noninstructional Interventions 172 Feedback Options

173

Job Performance Aids 176 Reward Systems 179 Employee Selection Practices

181

Organizational Redesign 184 Justifying Appropriate Noninstructional Interventions 187 Creating Design Specifications for Noninstructional Interventions 188

165

x

Contents

11 Selecting or Modifying Existing Instructional Materials 189 Selecting, Developing, or Modifying Instructional Materials 189 Conducting Cost-Benefit Analyses to Decide Whether to Use or Modify Existing Materials 194 Validating Selection or Modification of Existing Instruction 195

12 Developing Instructional Materials 196 Developing Media Specifications

196

Producing Instructional Materials in Various Delivery Formats 213 Developing Materials Aligned with Content Analysis, Proposed Technologies, Delivery Methods, and Instructional Strategies 214 Collaborating with Production Specialists 214

13 Designing Learning Assessments

215

Introduction 215 An Overview of Steps in Preparing Instruments 221 Constructing Reliable and Valid Methods of Assessing Learning and Performance 228 Ensuring the Assessment Is Aligned with Instructional Goals, Anticipated Learning Outcomes, and Instructional Strategies 229

PART FOUR: EVALUATING AND IMPLEMENTING INSTRUCTIONAL DESIGN 231 14 Evaluating Instructional and Noninstructional Interventions 233 Purpose and Definitions of Evaluation Formative Evaluation

234

239

Developing a Formative Evaluation Plan

240

Four Major Approaches to Conducting Formative Evaluation 246 Summative Evaluations

251

Reporting Evaluation Results 259 Creating the Report 261 Disseminating the Report 262

Contents

xi

15 Revising Instructional and Noninstructional Solutions Based on Data 263 Types of Revisions

264

Gaining Stakeholder Support for Revisions 267 Implementing Revisions to Delivery of Products/Programs 269

16 Implementing Instructional and Noninstructional Interventions Creating a Vision for Change

271

272

Planning for the Implementation and Dissemination of the Intervention 276 Planning for Diffusion of the Intervention 285 Encouraging and Achieving Adoption and Buy-In 285 Compliance versus Gaining Commitment 286 Monitoring Implementation, Dissemination, and Diffusion to Identify Potential Adjustments 287 Taking Action on Deviations or Problems Surfaced through Monitoring 288

PART FIVE: MANAGING INSTRUCTIONAL DESIGN

289

17 Applying Business Skills to Managing the Instructional Design Function 291 Where Instructional Design Fits into the Larger Organization 292 Contributing to the Organizational Purpose and Strategy 295 Implementing Standards 299 Leveraging Internal and External Resources to Accomplish Project Goals and Objectives 301 Managing Instructional Design Talent

303

Marketing and Communicating about Instructional Design 313 Navigating Organizational Culture, Structure, Politics, and Processes 314

18 Managing Partnerships and Collaborative Relationships 317 Identifying and Engaging Stakeholders

318

Building Relations between Instructional Designers and Individuals or Group Members 322

xii

Contents

Managing Cross-Functional Teams 327 Securing Commitment 331 Conducting Project Reviews

333

19 Planning and Managing Instructional Design Projects

334

Unique Challenges Posed by Project Management 335 Allocating Resources to Support the Project Plan 338

PART SIX: PREDICTING THE FUTURE OF INSTRUCTIONAL DESIGN 20 Predicting the Future of Learning and Instructional Design

341 343

Technological Advances to Improve Delivery and Design 343 The Impact of Neuroscience Research on Instructional Systems Design 357 A Final Word 361

About the Authors References Index

383

367

363

PREFACE TO THE FIFTH EDITION

I

nstructional design has changed dramatically since the fourth edition of this book was published in 2008. Those changes were just as dramatic as what happened between the first edition of this book in 1992, its second edition in 1998, and its third edition in 2004. Managers are trying harder than ever to improve human performance. In some respects, this pursuit of productivity improvement remains the modern business world’s equivalent of the medieval quest for the Holy Grail. There are good reasons for improving performance. The competitive environment has never been more fierce. The outsourcing of products and services—including instructional design work—is taken for granted in many settings. And there is increasing willingness to outsource or offshore services and manufacturing. Blended learning has grown popular, and it has led many instructional designers to feel like they need to relearn their profession from scratch. Then it, too, has undergone dramatic reformation, transformed into unique learning blends that include social media and m-(mobile-) learning. But one thing has not changed: the long-term success of organizations continues to hinge on improved human performance. That is important to remember even as new forms of technology are used, and experimented with, in design and delivery. Media use remains only the means to the end of improved performance, and instructional designers are still subject to the pointed complaint xiii

xiv

Preface to the Fifth Edition

by managers that they are too obsessed with media and not interested enough in achieving learning results that align with achieving their business strategies. For some, efforts to improve human performance have led to headlong plunges into flashy fads and quick fixes—modern-day snake-oil remedies. Others, however, are meeting the challenges they face by taking a more thoughtful approach: supporting and participating in the systematic analysis of human performance problems, identifying the root causes of those problems, finding and leveraging organizational and individual strengths, considering various improvement strategies, and carrying out the strategies in ways designed to reduce the unintended consequences of action. The desire is keen to accelerate this process through new models and approaches to instructional design, but many people have grown wiser (and more wary) about what it takes to make change happen, even as they have tried to do things faster. Instructional designers are often the standard-bearers of these calculated efforts to improve human performance. They continue to work under the guise of many job titles. They are sometimes called talent developers, performance technologists, performance consultants, trainers, training and development specialists, workplace learning and performance professionals, learning and performance professionals, instructional developers, staff development specialists, performance consultants, or instructional designers. A 2015 study revealed that over 30 percent of respondents to a global survey felt that the job title “instructional design” did not adequately reflect what they do—and one respondent felt that a better title was “jack of all trades” (Rothwell, Zaballero, Asino, Briskin, Swaggerty, and Bienert 2015). Whatever their titles, instructional designers share a common goal of improving human performance. Perhaps training remains as the best-known performance improvement strategy, although many people are experimenting with approaches that integrate planned learning, unplanned on-the-job learning, and social learning. Management solutions that do not include training range from using job aids, redesigning organizational structures and reporting relationships, redesigning work and jobs, refocusing employee selection methods, re-engineering workrelated feedback methods, and designing and implementing employee reward systems. Thousands of ways exist to improve human performance, and the sheer choice of strategies is as daunting as discovering root causes or building on strengths. The examples and references incorporated in this book grow out of our own experiences in instructional design. Since our experiences

Preface to the Fifth Edition

xv

have not been universal, we have attempted to focus on what we know. Our goal is to make this book a practical tool for instructional designers to develop their own skills and build their competencies in downto-earth ways. But one theme should be stressed from the outset: instructional systems design (ISD) is not about the mindless application of step-by-step schemes or new technology. Improving human performance is hard work. Both an art and a science, instructional design requires a blend of intuitive and analytical thinking. And it requires a willingness to meet needs to solve organizational problems, which (in turn) may demand that instructional designers skip steps in traditional instructional design models, multitask to do several steps at once, rearrange steps, add steps (such as translation), or even reinvent design models to meet the unique needs of unique clients in unique situations. The growing desire for accelerated approaches that align with the fast-paced demands of new technology has prompted new interest in diverse models to guide instructional design. The Foundation of the Book In the past, books on instructional design have often reflected the personal opinions of the authors and have not been based on an underlying foundation of solid research. However, Mastering the Instructional Design Process: A Systematic Approach, Fifth Edition, is based on Koszalka, Russ-Eft, and Reiser’s Instructional Design Competencies: The Standards (4th ed.) (2013). The fourth edition of this work is abbreviated throughout this book as The Standards. Prepared through the cooperation of highly respected professionals in the instructional design field, The Standards was sponsored by the International Board of Standards for Training, Performance, and Instruction (IBSTPI). The fifth edition of Mastering the Instructional Design Process: A Systematic Approach begins where The Standards leaves off. While The Standards focuses on what instructional designers do, this book focuses on how to demonstrate instructional design competencies. Its purpose is to point the way toward building and applying instructional design competencies. The Audience for the Book This book is for instructional design professionals and professionals in the making, whatever their formal job titles. It is intended as a desk aid to help professionals carry out their work and as a text for students. A list

xvi

Preface to the Fifth Edition

of references appears at the end of the book. We suggest that readers use it to pursue subjects of interest to them. Mastering the Instructional Design Process: A Systematic Approach, Fifth Edition, should also interest others, such as human resource professionals and operating managers, who have reason to analyze human performance problems systematically, pinpoint the root causes of those problems, identify strengths, consider various solutions to the problems or methods of building on strengths, and carry out the performance improvement efforts in ways designed to reduce the unintended side effects of action. Overview of the Contents. This book is adapted from The Standards. The chapters do not exactly follow the sequence of competencies in The Standards. That is necessary because print media is linear. Some may want to follow a step-by-step approach. But others, more wisely, will know to adapt their project approach to meet client and learner demands and project constraints. See a graphic illustrating the book’s organizational scheme in Exhibit P.1. The book opens with an Advance Organizer. The advance organizer helps readers assess their own grasp of instructional systems design.

Part One comprises only one chapter. Chapter One provides a conceptual foundation for the book. Its focus is on summarizing the instructional design field. This chapter sets the stage for the remainder of the book by

EXHIBIT P.1. AN INSTRUCTIONAL DESIGN MODEL Part VI: Predicting the Future of Instructional Design

Part I: Overview

Part V: Managing Instructional Design

Part II: Planning and Analyzing for Instructional Design

Part IV: Evaluating and Implementing Instructional Design

Part III: Designing and Developing for Instructional Design

Preface to the Fifth Edition

xvii

defining instructional design, describing instructional design as an emerging profession, and summarizing key issues affecting human performance in organizations. It also addresses several important critiques of traditional instructional design approaches. Part Two is entitled “Planning and Analyzing for Instructional Design.” It comprises Chapters Two through Six. Chapter Two is about conducting a needs assessment, Chapter Three is about collecting and analyzing data about instructional design projects, Chapter Four is about identifying the target population and environmental characteristics for instruction, Chapter Five is on using analytical techniques to determine instructional content, and Chapter Six is entitled “Analyzing the Potential Use of Existing and Emerging Technologies.” Part Three is called “Designing and Developing for Instructional Design.” Chapter Seven examines how to use an instructional design process appropriate for a project, Chapter Eight discusses ways to organize instructional programs and/or products, Chapter Nine reviews how to design instructional interventions, Chapter Ten reviews ways to plan noninstructional interventions, Chapter Eleven advises on ways to select or modify existing instructional materials, Chapter Twelve offers suggestions on developing instructional materials, and Chapter Thirteen focuses on how to design learning assessments. Part Four is called “Evaluating and Implementing Instructional Design.” It includes Chapters Fourteen through Sixteen. Chapter Fourteen suggests ways to evaluate instructional and noninstructional interventions, Chapter Fifteen examines ways to revise instructional and noninstructional solutions, and Chapter Sixteen gives advice on implementing instructional and noninstructional interventions. Part Five is “Managing Instructional Design.” Chapter Seventeen explains how to apply business skills to managing the instructional design function, Chapter Eighteen offers tips on managing partnerships and collaborative relationships, and Chapter Nineteen suggests how to plan and manage instructional design projects. Part Six is “Predicting the Future of Instructional Design.” It comprises Chapter Twenty, which concludes the book. It lists some trends that will shape the future of instructional design.

ACKNOWLEDGMENTS

W

e wish to thank members of the International Board of Standards for Training, Performance and Instruction (IBSTPI) for their encouragement of this project and their permission to use The Standards as the foundation for this book. (A description of IBSTPI appears on the next page for those who are curious about what the board is and what it stands for.) While any mistakes in this book are entirely our responsibility and not that of the International Board, we are indebted to the board members for their support. William J. Rothwell also thanks his graduate research assistant Jae Young Lee for her assistance in helping to secure, and repeatedly follow up on, necessary copyright permissions. William J. Rothwell University Park, Pennsylvania G.M. (Bud) Benscoter Raleigh, North Carolina

Marsha King Glenview, Illinois Steve King Glenview, Illinois

xix

ABOUT THE INTERNATIONAL BOARD OF STANDARDS FOR TRAINING, PERFORMANCE, AND INSTRUCTION

F

or more than three decades, The International Board of Standards for Training, Performance, and Instruction (IBSTPI® ) has been developing and validating the standards for professionals in the fields of training and performance. IBSTPI’s standards come about as a result of a rigorous research process, development, international validation, and publication. IBSTPI leads the way in developing standards for the training and performance professions. Worldwide, aspiring and active professionals, educational institutions, and organizations in a wide variety of sectors, including private industry, academia, military, and government, use IBSTPI’s standards to establish professional benchmarks, improve individual performance and organizational results, and to further extend the research about and around the competencies.

Source: Taken from www.linkedin.com/company/ibstpi. Used by permission of IBTSPI.

xxi

ADVANCE ORGANIZER

U

se the following instrument as an Advance Organizer for the book. Use it as a diagnostic tool to help you assess the need for improving your instructional design practices. You may also wish to use this instrument to refer directly to parts of the book that may be of special importance to you right now.

The Advance Organizer Directions: Read each item in the left column on the following pages. Spend about 10 minutes on this assessment. In the center column, circle a response that best fits your opinion: 0 = not applicable; 1= strongly disagree; 2 = disagree; 3 = neither agree nor disagree; 4 = agree; and 5 = strongly agree. Be honest! Think of instructional design as you believe you practice it. When you finish, add up the scores and then interpret them as shown at the end of this assessment. Be prepared to share your responses with others, perhaps your mentor(s), as a starting point for improving your instructional design practices. If you would like to learn more about an item, refer to the number in the right column to find the chapter in this book in which the subject is discussed.

xxiii

xxiv

Advance Organizer

Characteristics of Successful Instructional Design

How Well Do You Meet the Characteristic Based on Your Opinion? (Circle response below)

Chapter in the Book Covering the Topic

As an instructional designer, I effectively …

0

1

2

3

4

5

1. Define instructional design

0

1

2

3

4

5

1

2. Conduct a needs assessment

0

1

2

3

4

5

2

3. Collect and analyze data for instructional design projects

0

1

2

3

4

5

3

4. Identify the target population

0

1

2

3

4

5

4

5. Identify environmental characteristics influencing instruction

0

1

2

3

4

5

4

6. Use analytical techniques to determine instructional content

0

1

2

3

4

5

5

7. Analyze the potential use of existing technologies

0

1

2

3

4

5

6

8. Analyze the potential use of emerging technologies

0

1

2

3

4

5

6

9. Use an instructional design process appropriate for a project

0

1

2

3

4

5

7

10. Organize instructional programs and/or products

0

1

2

3

4

5

8

11. Design instructional interventions

0

1

2

3

4

5

9

12. Plan noninstructional interventions

0

1

2

3

4

5

10

13. Select instructional materials

0

1

2

3

4

5

11

14. Modify instructional materials

0

1

2

3

4

5

11

15. Develop instructional materials

0

1

2

3

4

5

12

16. Design learning assessments

0

1

2

3

4

5

13

17. Evaluate instructional interventions

0

1

2

3

4

5

14

18. Evaluate noninstructional interventions

0

1

2

3

4

5

14

19. Revise instructional and noninstructional interventions

0

1

2

3

4

5

15

20. Implement instructional and noninstructional interventions

0

1

2

3

4

5

16

Advance Organizer

xxv

Characteristics of Successful Instructional Design

How Well Do You Meet the Characteristic Based on Your Opinion? (Circle response below)

Chapter in the Book Covering the Topic

As an instructional designer, I effectively …

0

1

2

3

4

5

21. Apply business skills to manage the instructional design function

0

1

2

3

4

5

17

22. Manage partnerships and collaborative arrangements

0

1

2

3

4

5

18

23. Plan instructional design projects

0

1

2

3

4

5

19

24. Manage instructional design projects

0

1

2

3

4

5

19

25. Predict the future of instructional design projects

0

1

2

3

4

5

20

Total ………………………………………………… __________________

Scoring and Interpreting the Advance Organizer Total the scores above. Then interpret your score as follows:

Score

Interpretation

125–101

Grade your skills at the A level in instructional design. Excellent. Now plan for additional self-improvement.

100–76

Grade your skills at the B level in instructional design. Now plan for self-improvement in those areas you indicated needed additional improvement.

75–51

Grade your skills at the C level in instructional design. Average. Now plan for improvement.

50–26

Grade your skills at the D level in instructional design. Below average. Plan for improvement in those areas you indicated need. Get a mentor, listen to the advice you get, and act on it.

25–0

Grade your skills at the F level in instructional design. Far below average. Plan for improvement in all areas of instructional design. Get several mentors, listen to the advice they give you, and act on it. Failing that, you might consider a new career.

ABOUT THIS BOOK

Why Is This Topic Important? The term instructional design has two possible meanings. One meaning refers to a field of practice. A second meaning refers to ways to get good work results from human beings in organizational settings. Instructional design was invented to organize efficient, effective, systematic, and results-oriented training. In recent years, however, the term instructional design has gained a broader meaning associated with the myriad ways by which to achieve improvement in human performance. Those ways transcend mere training. But a key focal point of interest for many instructional designers remains how to improve human performance in organizational settings through effective training.

What Can You Achieve with This Book? How do you troubleshoot problems with human performance? When problems stem from individuals who lack knowledge, skill, or attitude to carry out the work effectively, how do you establish rigorous training that gets results? This book is primarily, though not exclusively, about xxvii

xxviii

About This Book

how to get results from training. It is based on research by seasoned practitioners on what it takes to analyze human performance problems and then—when training is an appropriate solution—to design and develop effective training, implement that training, and evaluate it.

How Is This Book Organized? This book is divided into seven distinct parts. Part One is an overview, describing instructional design. Part Two focuses on planning and analyzing for instructional design. Part Two also describes how to conduct a needs assessment, identify the targeted learner and environmental characteristics that affect instructional design projects, examine content to be used in instructional design, and how to examine the potential application of instructional technologies. Part Three reviews the design and development of instructional design projects. It reviews how to use an instructional design process appropriate for a project, organize and design instructional and noninstructional interventions, select or modify existing instructional materials, develop materials, and design learning assessments. Part Four centers on evaluating and implementing instructional design. It describes how to evaluate interventions, revise interventions, and implement interventions. Part Five summarizes how to manage instructional design, focusing on how to apply business skills to manage instructional design functions, manage partnerships, and plan and manage instructional design projects. Part Six, the final part, offers predictions for the future of instructional design.

How Is the Fifth Edition Different from the Fourth Edition? This book is updated with many new references and new sections on current issues in instructional design. It is based on the current edition (4th ed.) of Instructional Designer Competencies: The Standards (Koszalka, Russ-Eft, and Reiser 2013) and is also informed by the ASTD Competency Study: The Training and Development Profession Revisited (Arneson, Rothwell, and Naughton 2013) and by Skills, Trends, and Challenges in Instructional Design (Rothwell, Zaballero, Asino, Briskin, Swaggerty, and Bienert 2015).

Mastering the Instructional Design Process

PART ONE

OVERVIEW

Part VI: Predicting the Future of Instructional Design

Part I: Overview

Part V: Managing Instructional Design

Part II: Planning and Analyzing for Instructional Design

Part IV: Evaluating and Implementing Instructional Design

Part III: Designing and Developing for Instructional Design

CHAPTER ONE

AN OVERVIEW OF INSTRUCTIONAL DESIGN

I

nstructional design means more than literally creating instruction. It is associated with the broader concept of analyzing human performance problems systematically, identifying the root causes of those problems, considering various solutions to address the root causes, leveraging organizational and individual strengths, and implementing the interventions in ways designed to minimize the unintended consequences of action. Instructional design encompasses the preparation of work-related instruction and other strategies intended to improve worker performance. It does not mean throwing training at all problems or mindlessly plugging content into virtual templates. As we use the term, instructional design is (1) an emerging profession, (2) focused on establishing and maintaining efficient and effective human performance, (3) guided by a model of human performance, (4) carried out systematically, (5) based on open systems theory, and (6) oriented to finding and applying the most cost-effective solutions to human performance problems and discovering quantum leaps in productivity improvement through human innovation. We follow the International Board of Standards for Training, Performance, and Instruction (IBSTPI) by making basic assumptions about instructional design and competencies associated with it. (See Exhibit 1.1.) In this chapter, we will explore each of the six characteristics identified above to lay the groundwork for the 3

4

Mastering the Instructional Design Process

EXHIBIT 1.1. TEN KEY ASSUMPTIONS ABOUT INSTRUCTIONAL DESIGN AND INSTRUCTIONAL DESIGN COMPETENCIES • Assumption 1: Instructional designers are those persons who demonstrate design competencies on the job regardless of their job title or training. • Assumption 2: Instructional design (ID) competencies pertain to persons working in a wide range of settings. • Assumption 3: Instructional design is a process most commonly guided by systematic design models and principles. • Assumption 4: Instructional design is most commonly seen as resulting in transfer of training and organizational performance improvement. • Assumption 5: Instructional design competence spans novice, experienced, and expert designers. • Assumption 6: Few instructional designers, regardless of their levels of expertise, are able to successfully demonstrate all ID competencies. • Assumption 7: ID competencies are generic and amenable to customization. • Assumption 8: ID competencies define the manner in which design should be practiced. • Assumption 9: ID competencies reflect societal and disciplinary values and ethics. • Assumption 10: ID competencies should be meaningful and useful to designers worldwide. Source: R. Richey, D. Fields, and M. Foxon, Instructional Design Competencies: The Standards, 3rd ed. (Syracuse, NY: ERIC Clearinghouse on Information and Technology, 2001), 36–42. Copyright 1993 by the International Board of Standards for Training, Performance and Instruction. All rights reserved. Used with permission.

remainder of the book. We shall also address important critiques of traditional instructional design approaches.

Instructional Design: An Emerging Profession Instructional design is an emerging profession. People can—and do— enter jobs as instructional designers and work in that capacity for their entire careers. That is especially true when many organizations are converting their training to online, blended, and e-learning-based approaches. Instructional designers are often tasked to lead or facilitate such projects. Employment advertisements for instructional designers and closely aligned jobs frequently appear online and in print. (See, for instance, the job search websites run by the International Society for Performance Improvement at www.ispi.org and by the Association for Talent Development at www.astd.org.)

An Overview of Instructional Design

5

Many organizations across a broad spectrum of industries employ instructional designers. Jobs bearing this title are often positioned at the entry level. They often occupy the first rung on a career ladder leading to such higher-level jobs as instructor, project supervisor of instructional design, and Chief Learning Officer (CLO). But variations of this career ladder exist. Job titles also vary. Alternative job titles may include talent developer, performance technologist, performance consultant, human performance improvement specialist, human performance enhancement professional, instructional developer, education specialist, educational technologist, employee educator, trainer, staff development specialist, instructional technologist, or instructional systems specialist. Because variations exist in work duties, in modes of occupational entry, in educational preparation, and in career paths, instructional design is an emerging, rather than an established, profession. It is called a field of practice, though it has been researched (see Rothwell, Zaballero, Asino, Briskin, Swaggerty, and Bienert 2015). However, the trend has been toward certification in the field. That trend suggests increasing professionalism. For instance, the International Society for Performance Improvement supports a program leading to the Certified Performance Technologist (CPT) designation (for a description, see www.certifiedpt.org/WhatisCPT.htm). That follows a growing trend for certification of many kinds, ranging from individual (such as the CPT) to product or process accreditation (see www.iacet.org). ATD also offers the Certified Performance and Learning Professional (CPLP) designation to certify practitioners in a broad range of areas of expertise in the field (see www.td.org/Certification).

Instructional Design: Focused on Establishing and Maintaining Efficient and Effective Human Performance The chief aim of instructional design is to improve employee performance and to increase organizational efficiency and effectiveness. Instructional designers should be able to define such important terms as performance, efficiency, and effectiveness. What Is Performance? Performance is perhaps best understood as the achievement of results, the outcomes (ends) to which purposeful activities (means) are directed (see

6

Mastering the Instructional Design Process

Rothwell 2015; Rothwell, Benscoter, Park, Woocheol, and Zaballero 2014). It is not synonymous with behavior, the observable actions taken and the unobservable decisions made to achieve work results. However, behavior can contribute to results and is therefore important in considering those results. There are several types of performance. Human performance results from human skills, knowledge, and attitudes. Machine performance results from machine activities. Capital performance is about financial results. Company performance results from organizational activities. When asked to think about performance, most people in the United States think first of individual performance. There are at least two reasons why. First, people are sensitized to appraisals of individual performance because most organizations make evaluating performance an annual ritual, often linked to pay decisions. This practice has made a lasting impression on nearly everyone. Second, U.S. culture has long prized rugged individualism, implying that little lies beyond the reach of determined heroes exerting leadership and acting alone. However, continuing trends point toward a sustained emphasis on teams, groups, departments, divisions, or organizations. Those trends are as evident in the instructional design field—where team-based, and often virtual team-based, instructional design is becoming more commonplace—as in other fields. That trend is likely to continue as global virtual teams work continuously, and often through collaborative software, to design instruction and then deliver it through trainers. Defining Efficiency and Effectiveness Traditionally, two aspects of performance have been considered— efficiency and effectiveness. These terms have no universally accepted definitions. However, efficiency is usually understood to mean the ratio between the resources needed to achieve results (inputs) and the value of results (outputs). Some have said that the central question of efficiency can be posed as this: Are we doing things right? In this question, the phrase “doing things right” means “without unnecessary expenditures of time, money, or effort.” Effectiveness usually means the match between results achieved and those needed or desired. Its central question is this: Are we doing the right things? In this question, the phrase “right things” typically means “the results others, such as customers or key stakeholders, expect or need from the organization, group, or individual.”

An Overview of Instructional Design

7

Instructional Design: Guided by a Model of Human Performance Instructional design is guided by a model of human performance. In the most general sense, a model is a simplified or abstract representation of a process, device, or concept. A model of any kind helps understand a problem, situation, process, or device. It provides a basis for a common understanding, and common labels, for people to discuss the issue. This applies to a model of human performance, which is a simplified representation of factors involved in producing work results. It provides labels to key factors involved in performance and clues to pinpointing underlying causes of human performance problems. Many human performance models have been constructed (Abernathy 2010). They can be categorized as comprehensive or situation-specific. A comprehensive performance model includes factors affecting human performance in organizational settings. An example is shown in Figure 1.1 and Table 1.1. Table 1.1 defines and briefly describes the factors appearing in Figure 1.1. Rummler’s classic model, though published years ago, remains useful in analyzing human performance problems. (See Figure 1.2.) The root cause of the problem must be determined, and each factor in this simple model can be examined as a possible root cause. If it is not clear when the desired performance is necessary, the cause stems from the job situation. If workers cannot perform, the cause stems from the performers. If performers lack the skills or tools or other resources, the cause stems from the response (behavior). If the consequences of performing are punishing or do not exist, the cause of the problem stems from the consequences. If performers are given no information about the value of their performance, then the problem’s cause stems from inadequate or nonexistent feedback. Instructional designers should base what they do on a human performance model. Applying such a model to problem solving is the foundation of instructional design. The field is associated with analyzing human performance problems systematically, identifying the root cause or causes of those problems, considering various solutions to address the root causes, and implementing the solutions in ways designed to minimize the unintended consequences of corrective action. The logic is akin to that of a medical doctor who identifies symptoms, discovers underlying root causes of those symptoms, and then prescribes medicine or therapy to address the underlying causes.

8

Mastering the Instructional Design Process

FIGURE 1.1. A COMPREHENSIVE MODEL OF HUMAN PERFORMANCE IN ORGANIZATIONS ORGANIZATIONAL PERFORMANCE INDIVIDUAL PERFORMANCE Job context Motivation Knowledge Skills Attitudes Abilities Aptitude WORK-GROUP PERFORMANCE Structure Leadership Cohesiveness Roles Norms Status

Environment Structure Technology Strategy Culture

Source: Taken from W. Rothwell and H. Kazanas, Mastering the instructional Design Process: A Systematic Approach, 4th ed. (San Francisco: Pfeiffer, 2008), 7.

Another view has emerged in recent years. Some have criticized instructional design as too reactive, focused too much on solving problems rather than avoiding them or (better) building on organizational and individual strengths. An important goal is to establish an engagement culture (Rothwell, Alzhahmi, Baumgardner, Buchko, Kim, Myers, and Sherwani 2014). That requires discovering what is going right and what is best and then inspiring a dream of a better future, leading to a high-performance workplace where people are so engaged that they do not experience productivity problems (Richman and Kirlin 2015).

An Overview of Instructional Design

9

TABLE 1.1. FACTORS AFFECTING PERFORMANCE

Factor

Questions to Consider about the Influence of the Factor on Performance

Brief Definition

Individual Performance Job context

Motivation Knowledge

Skills Attitudes Abilities

Aptitude

The environment of the job, including supervisor(s), equipment and tools to be used, customers, and co-workers. The desire to perform. Facts and information essential to performing a job or task. Abilities to do things associated with successful job performance. Feelings about performance that are voiced to other people. Present capabilities to behave in certain ways. The future capability to behave in certain ways.

Do people have the necessary equipment, tools, and resources to perform? Do people want to perform? Do people have the necessary facts and information they need to take action and make decisions? Can people do the things associated with successful job performance? How do people feel about their behavior? Do people possess the necessary talents and mental or physical characteristics? Are people physically and/or mentally capable of learning how to perform?

Work-Group Performance Structure

The way work is allocated to members of a work group.

Leadership

The way directions are given to members of a work group.

Cohesiveness

The extent to which members of a work group are unified, pulling together as a group. The pattern of expected behaviors and results of each member of a group. Accepted beliefs of the work group.

Roles

Norms

Is responsibility for results clearly assigned? Are people aware of what they are responsible for? Are they held accountable for achieving results? Is it clear who is in charge? Does the leader consider how people feel (attitudes) as well as what must be done to achieve results (tasks)? Are people willing to work together to achieve desired results? Do members of a group understand what they are responsible for doing? How do members of a work group feel about the results they are to achieve? What methods are used to achieve those results?

10

Mastering the Instructional Design Process

TABLE 1.1. FACTORS AFFECTING PERFORMANCE, cont’d. Questions to Consider about the Influence of the Factor on Performance

Factor

Brief Definition

Status

The relative position of people in a group.

Do people have the formal authority to act in line with their responsibilities? Are other people willing to follow the lead of those who know what to do?

Organizational Performance Environment

The world outside the organization.

Structure

The way work is divided up and allocated to parts of the organization. Actions taken by people to change objects, people, or situations. Often refers to “how the work is done.”

Technology

Strategy

Culture

The means to achieve desired ends. It denotes an organization’s long-term direction. Beliefs and attitudes shared by members of an organization.

How well is the organization adapting to—or anticipating—changes outside it that affect it? Is work divided up appropriately?

Is the organization applying work methods that reflect current information about how to do the work? Is the organization competing effectively? Do members of the organization share common beliefs and attitudes about what they—and the organization—should do?

FIGURE 1.2. A SITUATION-SPECIFIC MODEL OF HUMAN PERFORMANCE Job Situation

Individual Performer

Response (Action or Decision) by Performer

Consequences of Action or Decision to Performer

Feedback Source: G. Rummler, “The Performance Audit,” in Training and Development Handbook: A Guide to Human Resource Development, 2nd ed., ed. R. Craig (New York: McGraw-Hill, 1976), 14-3. Reproduced with the permission of McGraw-Hill, Inc.

An Overview of Instructional Design

11

Instructional Design: Carried Out Systematically Instructional design is not just a field. It may also be a process for examining human performance problems and identifying solutions. The process should not be carried out intuitively; rather, its success depends on systematic application. Instructional designers place their faith in an iterative and systematic process that, viewed holistically, is more powerful than any single part. That process is not necessarily linear or step-by-step. Many systematic instructional design models have been constructed to guide instructional designers in their work.

Instructional Design: Based on Open Systems Theory Instructional design is based, in part, on open systems theory. An open system receives inputs from the environment, transforms them through operations within the system, submits outputs to the environment, and receives feedback indicating how well these functions are carried out. To survive, any open system must gain advantages from its transactions with the environment. Inputs include raw materials, people, capital, and information. Operations are activities within the organization that add value to raw materials. Outputs are services or finished goods released into the environment by the organization. Figure 1.3 illustrates these basic components of an open system. All open systems share common characteristics. First, they depend on the external environment for essential inputs and reception of their outputs. Second, there is a pattern to the flow of inputs and outputs. Third, all but the simplest open systems are composed of subsystems and interact with environmental suprasystems. A subsystem is a system within a system. A suprasystem is an overarching system that includes more than one system. As Katz and Kahn (1978) explain in their classic (and still relevant) treatment of open systems theory, most organizations comprise four generic subsystems. (They are called “generic” because they are found in most organizations, regardless of industry or reporting relationships.) The first is the production subsystem, which focuses on getting the work out. The second is the adaptive subsystem that includes any functions concerned with helping the organization change its internal operations to adapt to external environmental change. The third is the maintenance subsystem, which is

12

Mastering the Instructional Design Process

FIGURE 1.3. THE BASIC COMPONENTS OF AN ORGANIZATION AS AN OPEN SYSTEM ENVIRONMENT

Inputs

Operations

Outputs

Feedback

Source: Taken from W. Rothwell and H. Kazanas, Mastering the Instructional Design Process: A Systematic Approach, 4th ed. (San Francisco: Pfeiffer, 2008), 11.

concerned with streamlining internal operations and increasing efficiency. The fourth and last is the managerial subsystem, concerned with directing and coordinating the other three subsystems. Although organizations vary, in most firms the production or operations department exemplifies the production subsystem, the marketing department exemplifies the adaptive subsystem, the human resources department exemplifies the maintenance subsystem, and the top management team exemplifies the managerial subsystem. Most organizations function within many suprasystems. Perhaps the most obvious is the industry suprasystem, composed of all organizations involved in the same basic work. There are also other, equally important, suprasystems. The governmental-legal suprasystem, for instance, comprises all government agencies regulating the industry of which one organization is part. It also includes the laws, rules, and regulations with which the organization must comply. The marketing competitive suprasystem comprises all competitors, present and future. The economic suprasystem comprises the national and international economic environment within which the organization functions. The technological suprasystem is composed of the tools, state-of-the-art know-how, and work methods used in delivering the organization’s services or producing goods. The supplier suprasystem comprises all suppliers providing inputs to an organization. Each suprasystem influences organizational performance.

An Overview of Instructional Design

13

Open systems theory is important to instructional designers for two reasons. First, instructional designers recognize the critical importance of adapting to, and even anticipating, changes in the environment. Organizational and individual effectiveness depends on how well work results match environmental demands. Hence, one question that should be asked in any performance improvement effort is this: How much will this project contribute to the organization’s ability to adapt to changing environmental conditions? If the answer is “not much” or “we don’t know,” then maybe performance improvement activities should be directed to other projects. Second, instructional designers recognize that any corrective action taken to change one subsystem will affect others. The parts of any organization (system) are as interdependent as the strands of a spider web. A change in one part will affect others, just as an entire spider web vibrates when one strand is disturbed. If a change is made in the people selected into a job category, it will affect the training they should receive. Large system changes in organizations will have effects partially predictable—and partially unpredictable. However, order exists even amid apparent random disorder, a central view held by advocates of complexity theory. Observers of the instructional design field have repeatedly emphasized that much can be learned from complexity theory. Complexity theory enriches the traditional open systems orientation by providing a holistic view, rife with unpredictability, to the instructional design process (Johnson 2010).

Instructional Design: Oriented to Finding and Applying the Most Cost-Effective Solutions to Human Performance Problems Instructional designers sometimes assume, mistakenly, that their role is to “offer job-oriented instruction”—which means “training.” Sometimes others in the organization share the same misconception of their role. Human performance problems are complex and cannot always be solved by simplistic solutions such as instruction alone. Instruction should only be used when the performance problem stems from a lack of knowledge or skills or the wrong attitudes and when instruction is the most cost-effective solution. Since we will use the terms knowledge, skills, and attitudes throughout this book, perhaps some definitions are in order. Knowledge refers to

14

Mastering the Instructional Design Process

what people must know to do their work; skills are associated with the ability to perform; and attitudes center on what people feel about what they do. As work becomes more focused on deciding, processing information, and servicing customers, attitudes—traditionally neglected by instructional designers in favor of knowledge and skills—are becoming more important in the mix of what leads to effective performance. Instruction should not be the solution when a performance problem stems from lack of motivation, feedback, incentives, or some other cause. It is also a costly solution because it demands substantial investments of time and money to prepare effective instructional materials, test them, revise them, deliver them, and evaluate them. Employees receiving off-the-job instruction lose time doing work and are usually paid while learning, which adds to the cost. Instructional designers and others involved in preparing instructional materials must be paid, which further adds to the cost. For all these reasons, work-oriented instruction is a costly way to improve performance. It should be a solution of last resort. Instructional designers should be certain there will be a favorable return on any investment in, and real business impact for, performance improvement efforts. They may apply many methods of cost-benefit forecasting and analysis to estimate the expected return (payoff) on the investment. First they estimate the cost of the performance problem. Then they estimate the expected costs to rectify the problem. Finally, they compare the two. If a return on investment takes too long, instructional designers should direct their attention to other projects in which the benefits are more certain, payoffs are higher, or results can be achieved faster. What it takes to be an effective instructional designer today can be daunting. Many studies, apart from The Standards, have been conducted on instructional design competencies or related topics (see Kaufman and Bernardez 2012; Klein and Jun 2014; Marker, Villachica, Stepich, Allen, and Stanton 2014; Reiser and Dempsey 2011; Stolovitch 2015; Sugar, Hoard, Brown, and Daniels 2011). Almost everyone agrees that the field is demanding—and is becoming more so—due to intense time and cost pressures.

Criticisms of Traditional Instructional Approaches No field of endeavor is immune to criticism. That is as true of instructional design as it is of any field. Critics of traditional instructional system design (ISD) approaches have grown increasingly strident in their

An Overview of Instructional Design

15

complaints about its real and perceived shortcomings. It is worthwhile to discuss early in this book the most serious concerns voiced. In a classic article that launched a continuing debate, Merrill, Li, and Jones (1990) distinguished between First Generation Instructional Design, which they designate ID1 , and Second Generation Instructional Design, which they designate ID2 . ID1 “assumes a cumulative organization of learning events based on prerequisite relationships among learned behaviors” (7). ID1 has long dominated the field but suffers from many limitations, according to the article’s authors. They believe it focuses on parts rather than integrated wholes, provides superficial advice for organizing instruction, adopts a closed-system view of instruction that disregards the environment in which instruction is carried out, asserts an unrealistic approach to instructional development, and produces instruction that is to learners passive (and boring) rather than active (and motivating). To solve these problems, the authors argued that a new ID2 paradigm is needed in the instructional design field. ID2 will lend itself to “analyzing, representing, and guiding instruction to teach integrated sets of knowledge and skills.” It will also suggest ways to select “interactive instructional strategies” and will be “an open system” that is “able to incorporate new knowledge about teaching and learning and to apply these in the design process.” In addition, ID2 should—among other innovations—“organize knowledge about instructional design and define a methodology for performing instructional design,” provide “a series of intelligent computer-based design tools for knowledge analysis/acquisition, strategy analysis and transaction generation/configuration,” and make use of “a collection of mini-experts, each contributing a small knowledge base relevant to a particular instructional design decision or set of such decisions” (Merrill, Li, and Jones 1990, 10). More recently, Merrill has recommended a “pebble in the pond” approach that relies on key principles to guide instructional design (Merrill 2002 & 2015). Other authorities in the instructional design field have joined the chorus calling for innovative new approaches to meet the daunting challenges facing today’s instructional designers. One central dilemma, however, may not be that the field is in need of new models to guide instructional design but that existing models are not applied. Additional critiques of the traditional ISD model have surfaced over many years. One complaint is about the process. It is, the critics contend, too slow and overly analytical for a frenetically paced world. The second complaint is about the practice. The ISD model is too linear, leading to an inflexible approach. ISD need not be treated that way—but, the critics

16

Mastering the Instructional Design Process

assert, it too often is treated that way. Third, technological innovations have rendered the ISD model out of touch. What may have worked for classroom-based training is not appropriate, or even desirable, for e-learning and many emerging instructional technologies. The emergence of social media, in which individuals can communicate in real time, leads to increased pressure for real-time, instant-messaging-style instruction. Some believe that the problem with e-learning and other forms of technologically dependent instruction, itself under attack, is its tendency to truncate necessary steps of analysis, design, development, implementation, and evaluation. The result is that sometimes, critics contend, instruction is thrown at problems it can never solve because management action is needed instead. In recent years, much attention has focused on the SAM (which stands for Successive Approximation Model) model as an alternative to ISD (Allen 2012). SAM is based on the notion, usually associated with the engineering field, of rapid prototyping. Rapid prototyping assumes that any organizational effort will be fraught with mistakes. The goal of SAM is to learn from mistakes quickly and get instruction out faster. The motto of SAM could be “get some instruction out there, assume it will be flawed, test it, and then move through overcoming the flaws as quickly as possible.” Given the fast pace of many organizations, SAM responds to the feeling that the ISD model takes too long, even though critics call it a “shoot first and aim later” approach. Other instructional designers eschew all models and just try to survive. They make it up as they go along. They may apply a model such as ISD or SAM or else draw from models in their own idiosyncratic approaches to design. While the danger of a purely ad hoc approach is that much will be forgotten, a games-based approach or an eclectic approach enjoys the benefit of flexibility in the face of daily challenges for speed and results (Kapp 2012; Rothwell, Zaballero, Asino, Briskin, Swaggerty, and Bienert 2015). But one thing is clear: there is considerable pressure to reduce the time to deliver effective learning experiences. If the instructional design process appears to be slow and ponderous—which it does not have to be—the pressure is on to slash through slow turnaround times and experiment with more rapid, yet still effective, approaches to instructional design. Our goal in the following chapters is to describe the competencies of instructional design work and provide the means by which practitioners can develop, or sharpen, their abilities.

PART TWO

PLANNING AND ANALYZING FOR INSTRUCTIONAL DESIGN

Part VI: Predicting the Future of Instructional Design

Part I: Overview

Part V: Managing Instructional Design

Part II: Planning and Analyzing for Instructional Design

Part IV: Evaluating and Implementing Instructional Design

Part III: Designing and Developing for Instructional Design

CHAPTER TWO

CONDUCTING A NEEDS ASSESSMENT

I

n this chapter, we define terms associated with needs assessment, describe how to distinguish between a learning and a performance problem, describe essential steps in developing needs assessment plans, review typical problems likely to arise during needs assessment, suggest ways of overcoming these problems, explain how to identify instructional and noninstructional problems based on needs assessment results, identify approaches to estimating costs and benefits of possible solutions, describe the preparation and dissemination of a needs assessment report, provide a simple case study highlighting important issues in needs assessment, offer advice on judging and justifying needs assessment, address key ethical and cross-cultural issues in conducting needs assessment, and review recent developments in needs assessment. According to The Standards (Koszalka, Russ-Eft, and Reiser 2013, 24–30), one competency for instructional design is to “conduct a needs assessment in order to recommend appropriate design solutions and strategies.” It is an advanced competency. The performance statements associated with this competency indicate that instructional designers should be able to: “(a) identify varying perceptions of need among stakeholders and the implications of those perceptions (advanced); (b) describe the nature of a learning or performance problem (essential);

19

20

Mastering the Instructional Design Process

(c) determine the root causes of identified discrepancies (advanced); (d) synthesize findings to identify and recommend potential instructional and noninstructional solutions (advanced); (e) estimate costs and benefits of possible solutions (advanced); and (f) prepare and disseminate a needs assessment report (advanced). The needs assessment process is the first step in the traditional instructional systems design process. Needs assessment is important because all subsequent steps in the instructional systems design (ISD) model depend on its results (Rothwell and Sredl 2000). It is depicted in a circular fashion (Figure 2.1) as a reminder that the entire process is iterative; that is, you can return to and revise any step in the process based on what you’ve discovered in subsequent steps.

FIGURE 2.1. A MODEL OF STEPS IN THE INSTRUCTIONAL DESIGN PROCESS Conduct a needs assessment

Evaluate instruction

Assess relevant characteristics of learners

Design instructional materials

Analyze characteristics of a work setting

Specify instructional strategies

Perform job, task, and content analysis

Sequence performance objectives

Write statements of performance objectives

Develop performance measurements

Source: W. Foshay, K. Silber, and O. Westgaard, Instructional Design Competencies: The Standards (Iowa City: IA: International Board of Standards for Training, Performance, and Instruction, 1986), 3. Copyright © 1993 by the International Board of Standards for Training, Performance, and Instruction. All rights reserved. Used with permission.

Conducting a Needs Assessment

21

As has been noted earlier in this publication, instructional designers are now expected to apply a process that identifies needs that can be met with either instructional or noninstructional solutions. The needs assessment process in today’s world should be conducted without a preconceived bias toward a training solution. In the past, an instructional designer would normally perceive his or her role as conducting a training needs assessment. Today, however, that same individual is accountable for conducting a performance needs assessment, which greatly expands the needs assessment process. Before examining the steps required to conduct a needs assessment, several points should be emphasized related to the overall process. 1. The instructional designer/needs assessor and the client must be very clear about the purpose of the needs assessment. Sometimes clients will be less than candid about the purpose of conducting a needs assessment. An assessment may be conducted to justify a reduction in force or other purposes hidden to the assessor and the people who provide data for the assessment. This presents an ethical challenge for the instructional designer, making it mandatory that the assessor and client establish a contract that specifically states the purpose of the assessment and that any deviations from the contract be made only with the concurrence of both parties. 2. The term “needs assessment” may carry a negative implicit meaning for those who will be involved. Hannum (2013) points out that, since needs assessments uncovers discrepancies between what is and what should be, it implies that something is lacking in the organization or its employees. Hannum points out that several approaches to organizational improvement (appreciative inquiry, strength finders) focus on what an organization is doing well and attempt to build on everyone’s strengths. The needs assessment process can easily be positioned as a tool for this purpose. The language we use to describe the process becomes critical. 3. While the traditional ISD model has been criticized—and some say it is inappropriate for e-learning or other technologically based instruction—we believe it still serves as a reliable guide. That is a view others share (Beckshi and Doty 2000). The ISD model may be even more important with e-learning and with blended learning than it has been historically for classroom-based instruction (Barbian 2002). One reason is that designing effective e-learning can be costly and time-consuming, and it is important to be sure that problems to be solved really are suitable for an instructional solution. In addition,

22

Mastering the Instructional Design Process

several instructional design models specifically targeted to designing e-learning or web-based learning are available for today’s designers. The purpose of needs assessment is to uncover what the performance problem is, who it affects, how it affects them, and what results are to be achieved by training. However, daunting obstacles must be overcome to perform needs assessment successfully. One obstacle is the perception of top managers that instructional designers or trainers are self-interested parties in the needs assessment process who may stand to benefit from the results by justifying their existence (Bengtson 1994; Rothwell, Lindholm, and Wallick 2003). As the field embraces performance analysis as a skill set, however, this resistance should diminish, because the solution is not hampered by a training-only bias. A second obstacle is that needs assessment is sometimes thought to take too much time in an age of dynamic change. Allen (2012) points out that a good deal of time and effort is spent on identifying needs and solutions, often without producing the desired results. He notes that “this analysis is often incomplete and inaccurate—almost by necessity because so much ‘nonproductive’ time can be spent on it, relevant data are hard to acquire, available information is often misleading, and situations change while the analysis is underway” (6). A major challenge is balancing rigorous with speedy analysis—despite a growing array of online approaches to provide support for conducting needs assessment. To add value to the needs assessment process, we have to understand what it is and what it isn’t. Muller and Roberts (2010) describe seven myths related to needs assessment and the cure for each (as shown in Table 2.1). Wedman (2008) proposes a needs assessment approach he calls the Performance Pyramid. While it is supported by several analytical tools, Wedman emphasizes that needs assessment should be an organic, rather than a mechanistic, process. Close collaboration with your client will help you identify the “path of least resistance.” He suggests four layers to the process, each adding more value, time, and resources. Layer 1: “Fix it now.” This layer suggests a limited amount of time to complete an assessment, few resources and, perhaps, limited value in the outcome. Layer 2: “Fix it soon.” With this layer, more time, resources, and budget may be expended with a greater likelihood of value being achieved.

Conducting a Needs Assessment

23

TABLE 2.1. MYTHS AND CURES Myth

Cure

A training solution is always the cause of a performance problem Needs assessment takes too long to complete The client will never accept the needs assessment process

Conduct a thorough diagnosis to uncover what’s likely to be multiple sources of the problem.

The needs assessment process is too complex Needs assessment is too costly We don’t have a strong presence with executives Training is mandated: no time for a needs assessment

Looks for ways to reduce the needs assessment cycle. Collaborate with your client to understand the business better. Focus particularly on line managers. Work with line managers to clearly define the problem and identify the best sources of information. Choose the proper method based on your knowledge of the costs of each approach. Know the business well and talk in business (not training) language. Conduct an assessment during training for input into future decisions.

Source: Adapted from “Seven Cures to Skipping the Needs Assessment,” T&D 64 (March 2010): 3.

Layer 3: “Here’s your budget and deadline.” Budgets are predetermined and the needs assessor must work within these constraints. Layer 4: “Lots of time and $$$.” While a rare occurrence, the assessor has an expansive amount of time and resources to conduct a thorough needs assessment, achieving maximum value for the outcome. To conduct a thorough needs assessment, several terms should be agreed upon to avoid confusion with the process. Let’s define those terms before examining the process itself.

Defining Terms To understand needs assessment, instructional designers should first understand the meaning of key terms associated with it. Such terms include need, needs assessment, needs analysis, training requirements analysis, needs assessment planning, and the needs assessment plan itself.

24

Mastering the Instructional Design Process

Need A need has traditionally been defined as a performance gap separating what people know, do, or feel from what they should know, do, or feel to perform competently. The word “need” should be a noun, not a verb (Kaufman 1986). The reason: when need is a verb in the sentence, “We need some training on time management,” it implies something merely desirable (a want) rather than something essential to competent performance. A need should always be linked to the essential knowledge, skills, and attitudes an individual must possess to perform work competently and accomplish the desired results.

Needs Assessment A needs assessment “identifies gaps in results, places them in order of priority, and selects the most important for closure or reduction” (Watkins and Kaufman 1996, 13). It is undertaken to “identify, document, and justify gaps between what is and what should be and place the gaps in priority order for closure” (Kaufman 1986, 38). Although such gaps are a traditional starting point for developing instruction (Rothwell and Sredl 2000), analyzing gaps is sometimes fraught with the problem of overlooking the performance levels of experts in a group or organization (Lewis and Bjorkquist 1992). In a classic discussion, Kaufman and English (1979) identify six types of needs assessment arranged in a hierarchical order of complexity. The first, and least complex, is alpha assessment. It focuses on identifying a performance problem. (An alpha assessment is synonymous with performance analysis.) A beta assessment is the second type. It is based on the assumption that an employee performance problem exists but that alternative solutions must be weighed for their relative cost-benefit and practicality. A gamma assessment, the third type, examines differences between alternative solutions to a performance problem. The fourth type is a delta assessment, and it examines performance gaps between what is and what should be. An epsilon assessment, the fifth type, examines discrepancies between desired and actual results of an event. A zeta assessment is the sixth type. It involves continuous assessment and evaluation in which regular feedback is used to monitor solutions and make corrective changes if they are necessary.

Conducting a Needs Assessment

25

More recently, Kaufman and Guerra-Lopez (2013) emphasize that, from a performance perspective, a needs assessment can focus on several levels of organizational results, including: • The strategic level: The impact of external factors (legislation, competition) on the organization. • The tactical level: Examining the overall results of the organization (for example, market share, profitability). • The operational level: Examining internal deliverables and how effectively and efficiently they are produced. These various levels underscore what has become a well-known approach to needs assessment called the Organizational Elements Model, or OEM (Kaufman, Thiagarajan, and MacGillis 1997). In this approach, there are five levels at which one may conduct a needs assessment. The first three levels constitute “ends” (that is, products, outputs, impact) while the final two levels represent “means” (that is, how the work gets done): • The Mega Level: The results or outcomes that society experiences because of an organization’s performance. • The Macro Level: The results experienced by the organization itself; for example, sales, profit, market share. • The Micro Level: The internal results produced by the organization; for example, number of training programs, numbers of widgets produced, scrap and rework. • The Process Level: The efficiency and effectiveness of the organization’s internal processes; for example, hiring, training, marketing. • The Inputs Level: The assessment of the resources needed (for example, employees, capital, regulations) to operate the organization. From this discussion, we can see there are several approaches to conducting a needs assessment. Whichever approach is chosen, the next overall step, once the results of the assessment are known, is to conduct a needs analysis. Needs Analysis A needs analysis discovers the underlying causes of gaps between the ideal or desirable and the actual. It is usually carried out following a needs

26

Mastering the Instructional Design Process

assessment. It goes beyond a needs assessment, which merely shows that a performance gap exists, to pinpoint the root cause(s) leading to that gap. Identifying root causes is essential for discovering the best solutions (Rothwell, Benscoter, Park, Woocheol, and Zabellero 2014; Rothwell, Hohne, and King 2000). Barbazette (2006) identifies the following needs analyses: 1. Performance (or gap) analysis: Is the issue a skill or knowledge deficiency or something else? 2. Feasibility analysis: Is the benefit of providing a training solution greater than the cost of the solution? 3. Needs versus wants analysis: Is the deficiency linked to an actual business need or is it something that one or more stakeholders believe should be the solution? 4. Goal analysis: What is the specific behavioral improvement that will result from a “fuzzy” want or desire? 5. Job/task analysis: How should this job or task be done correctly and how do they break down into their component subtasks? 6. Target population analysis: What do we know about the audience to be trained and might there be other audiences who would benefit from the training? 7. Contextual analysis: What are the other requirements (that is, beyond the training content) that will influence the success of the training? If we have determined that the solution to the performance gap is a training solution, we are now prepared to identify the training requirements. Training Requirements Analysis Care should be taken to avoid making assumptions too quickly about the causes of gaps. As Watkins and Kaufman (1996, 13) point out, “Although the term ‘training needs assessment’ is popular in the field, it seems to be an oxymoron. If you know that training is the solution, why do a needs assessment? A more accurate label for a ‘training needs assessment’ is ’training requirements analysis.’ A training requirements analysis can be a useful and important approach to designing training that will respond to your needs after you have defined them.” A training requirements analysis (TRA) thus specifies exactly what training is necessary.

Conducting a Needs Assessment

27

Needs Assessment Planning Needs assessment planning means developing a blueprint for collecting needs assessment information. It should not be confused with a needs assessment plan. Planning is a process, while a plan is a product (Rothwell and Cookson 1997). For needs assessment planning to be handled successfully, key line managers and other interested groups should participate in each step of designing the needs assessment plan and interpreting the results. Participation in needs assessment, as in many organizational activities, is essential to building ownership among key stakeholders. In the broadest sense, needs assessment planning can be categorized into two types: comprehensive and situation-specific. Comprehensive needs assessment planning is broad, covering large groups inside or outside an organization (Rothwell and Kazanas 1994a). Sometimes called macro needs assessment (Laird 1985), it is appropriate for determining the continuous and relatively predictable training needs of all newly hired workers, since they must be oriented to their jobs. The results of a comprehensive needs assessment are used to establish an organization’s curriculum—an instructional plan—covering basic training for each job category. A curriculum provides long-term direction to organized learning activities (Rothwell and Kazanas 1994a, 2003; Rothwell and Sredl 2000). Situation-specific needs assessment planning is narrower. Sometimes called micro needs assessment (Laird 1985), it is appropriate for correcting a performance problem that affects only a few people. For instance, a micro training need exists when one supervisor reveals no knowledge of “progressive discipline” in firing an employee, but other supervisors possess that knowledge. More often than not, instructional designers devote their attention to situation-specific needs. There are several reasons. First, relatively few organizations establish an instructional plan or training curriculum across all job categories. They lose the advantages that could be gained by pursuing a long-term direction for instructional activities in the organization. Second, situation-specific needs often have built-in management support. Since the performance problem already exists, has visible symptoms, and affects an identifiable target group, instructional designers find they already have a constituency of interested stakeholders who are eager to support efforts—and furnish resources—to solve the problem. One primary duty of an instructional designer is to fully know about the instructional assets that exist inside and outside the organization (prior training programs, job-specific documentation, external training,

28

Mastering the Instructional Design Process

and technical references). The designer must be able to align these resources with the needs of the business. Gendelman (2009) underscores this alignment requirement and points out several advantages for recycling instructional assets: 1. Enhance the alignment between business needs and current instructional resources. 2. Increase department budgets by demonstrating the alignment of instructional assets to the needs of the business and its executives. 3. Decrease the training department’s costs for new development, content acquisition, and licensing. Gendelman (2009) proposes the following steps to ensure alignment between training assets and business goals: 1. Know your audience well, including prerequisites and motivational factors of the audiences you serve. Create a “Position Road Map” of the audiences you serve, indicating the tasks they perform, the outcomes of their jobs, desired behaviors, and job constraints. 2. Analyze the existing curricula and identify gaps between the assets in place and the ones you need to close the gaps. 3. Align the needs of the business with the curriculum assets already in place and any discrepancies that exist. 4. Make recommendations for instructional material that must be developed or acquired, any assets that must be retired, and those that can be reused. 5. Implement your recommendations, including acquisition of outside material, asset development, or other recommended actions. 6. Assess the effectiveness of your revised curriculum and revise as needed. Now that you’ve completed the needs assessment, it’s time to develop your needs assessment plan. The next section will introduce you to this process. Needs Assessment Plan A needs assessment plan is a blueprint for collecting information about instructional needs. A needs assessment plan assumes that sufficient justification already exists to solve a human performance problem. The plan usually resembles a research plan (a proposal for conducting a research study).

Conducting a Needs Assessment

29

A needs assessment plan should usually address seven key issues (Foshay, Silber, and Westgaard 1986, p. 27). 1. Objectives: What results are desired from the needs assessment? 2. Target audience: Whose needs will be assessed? 3. Sampling procedures: What methods will be used to select a representative group of people from the target audience for participation in the needs assessment? 4. Data collection methods: How will information about needs be gathered? 5. Specifications for instruments and protocols: What instruments should be used during needs assessment, and how should they be used? What approvals or protocols are necessary for conducting the needs assessment, and how will the instructional designer interact with members of the organization? 6. Methods of data analysis: How will the information collected during needs assessment be analyzed? 7. Descriptions of how decisions will be made based on the data: How will needs be identified from the results of data collection and analysis? The anecdotal evidence available about needs assessment consistently shows that such issues are given varying degrees of emphasis depending on project constraints and stakeholder requirements and expectations.

Steps in Developing Needs Assessment To develop a needs assessment plan, instructional designers should first clarify why they are doing the assessment. Beyond that, the place to start depends on the problem to be solved, the number of people affected by it, and the time span available for the intended solution. The starting point for an alpha needs assessment (identifying performance problems) is not the same as that for a delta assessment (examining performance gaps between what is and what should be). Likewise, the starting point for a comprehensive needs assessment differs from a situation-specific needs assessment. Instructional designers who set out to develop a plan for a comprehensive needs assessment adequate for establishing a long-term instructional plan for an organization or an employee job category are embarking on an ambitious undertaking akin to corporate strategic business planning. They should begin by locating a current organization

30

Mastering the Instructional Design Process

chart and information about strategic business plans, job categories in the organization, common movements from one job category to another (career path maps), existing human performance problems in each job category, and individual training needs. They should then identify, for each job category, the knowledge, skills, and attitudes for employees to perform competently. They should use the skills list as the basis for a curriculum by job category, team, department, or geographical site. Instructional designers who are developing a situation-specific needs assessment plan designed to close a performance gap through instruction should begin by clarifying what they know about it. (For example, see Table 2.2). Having answered these questions, instructional designers should then move on to establish objectives, identify the target audience, select sampling procedures, decide on data collection methods, specify instruments and protocols, choose methods of data analysis, and describe how decisions will be made based on the data. We now turn to a discussion of each step. Establishing Objectives of a Needs Assessment Needs assessment objectives spell out the results sought from needs assessment. In a written needs assessment plan, they should appear immediately after a succinct description of the performance problem to be investigated. Needs assessment objectives, much like instructional objectives, provide direction. They reduce the chance that instructional designers might get sidetracked studying tangential issues during the assessment process. In addition, they also clarify why the problem is worth solving and what the ideal assessment outcomes will be. To establish needs assessment objectives, instructional designers should begin by clarifying what results are to be achieved from the needs assessment. This is a visioning activity that should produce a mental picture of the desired conditions existing at the end of the assessment process. Once the vision has been formulated, instructional designers should then write a short (one- to two-page) proposal for conducting the needs assessment. This proposal should be a selling tool and submitted as a formal request. Most important, it should build ownership for the assessment among key decision makers. Results can be thought of in several ways. One desirable result of needs assessment is agreement among stakeholders about what the needs are and what instruction should meet them. A second desirable result is a

Conducting a Needs Assessment

31

TABLE 2.2. QUESTIONS FOR DEVELOPING A SITUATION-SPECIFIC NEEDS ASSESSMENT PLAN Question

Related Issues

What is happening now?

How are people presently performing? What results (levels of outputs and quality) are now being achieved? What are the relevant work standards or performance objectives? What relationship, if any, exists between the organization’s strategic business plan and employee performance? How do managers and others (such as customers) want people to behave or perform? What results should be achieved by employees? How much does management or other stakeholders (such as customers or shareholders) want this idea state to exist? How can the gap be measured? What historical trends are evident? Is the gap increasing over time? What effects (consequences) of the gap are evident in the organization? How does the gap affect individuals inside the targeted group? Outside the targeted group? Can the problem be broken down into parts? Are some parts of the problem (that is, subproblems) caused by deficiencies in knowledge, skills, or attitudes, while others are caused by deficiencies in the environment? How should subproblems caused by environmental deficiencies be solved? How should subproblems caused by deficiencies in knowledge, skills, or attitudes be solved? Will efforts to investigate problems or subproblems change them because people will modify their behaviors during the investigation process? Will data collection efforts create expectations, realistic or otherwise, about management actions or solutions? Will decision makers interpret results of needs assessment in conformity with logical conclusions reached, or will they impose their own personal interpretations on results?

What should be happening?

How wide is the performance gap between “what is” and “what should be?” How important is the performance gap?

How much of the performance gap is caused by deficiencies in knowledge, skills, or attitudes? What solutions are costeffective and feasible?

What unintended side effects of taking corrective action can be predicted?

Source: W. Rothwell, Beyond Training and Development, 3rd ed. (Amherst, MA: HRD Press, 2015). Used by permission of HRD Press.

32

Mastering the Instructional Design Process

sense of what learners must know, do, or feel to overcome the deficiency of knowledge causing the performance problem. A third and final result is a clear sense of the final work product of the needs assessment. By thinking about the final work product, instructional designers clarify just how the results should be presented to stakeholders. Should the needs assessment results be described in a detailed report, a memo, a letter, an executive briefing, an electronic mail message, a website, or some combination? Objectives can take different forms in a needs assessment plan. They can be presented as questions about a performance problem, statements of desired results, or statistically testable hypotheses. Questions are appropriate when the aim is to use information collected during needs assessment to stimulate organizational change. Statistically testable hypotheses are appropriate only when assessment will be carried out with extraordinary rigor and the information collected during assessment will be subjected to statistical analysis. Any legitimate resource on social science research will contain sections on establishing “research objectives,” a topic that can be readily translated into advice about preparing “needs assessment objectives.” Identifying the Target Audience Whose instructional needs are to be addressed in solving the performance problem? Who must be persuaded by the results of the needs assessment to authorize instructional projects and provide resources for carrying them out? To answer these questions, instructional designers must identify target audiences. Any needs assessment really has at least two target audiences: performers and decision makers. Performers are employees whose instructional needs will be identified through the needs assessment process. They correspond to subjects in a research project. Any needs assessment must identify who is affected by the performance problem, how much they are affected, and where they are located. In micro training needs assessment projects focusing on a single work unit, it may be possible to identify only a few individuals whose needs should be examined. But in most macro training needs assessment projects, it will be necessary to consider instructional needs by employee job categories or departments. Each job class may be viewed as a different market segment for instruction, and each segment may differ in needs. If human performance problems stem from lack of employee knowledge about such organizational “rules” as dress code or hours of work, employees may lack knowledge, while supervisors, managers, or team members

Conducting a Needs Assessment

33

may lack knowledge of how to deal with the corrective action stemming from those problems. Decision makers are the individuals whose support will be crucial if the needs assessment plan is to be carried out successfully. They may include instructional designers who will use results of the needs assessment and supervisors of employees who will receive instruction. It is essential to identify who will receive results of the needs assessment, because their personal values and beliefs will affect the interpretation of the results. Specifying Instruments and Protocols Data collection and analysis tools and procedures will be discussed in Chapter Six of this publication. There are very important considerations related to these tools that should be considered in developing a comprehensive needs assessment plan. This section of the chapter will focus on these two considerations. The needs assessment plan should answer the following questions: 1. What instruments should be used during the needs assessment, and how should they be used? 2. What approvals or protocols are necessary for conducting the needs assessment, and how will the instructional designer interact with members of the organization? These questions must be addressed in a needs assessment plan. The first has to do with specifying instruments, a topic covered in Chapter Six; the second has to do with specifying protocol. Protocol means diplomatic etiquette and must be considered in planning needs assessment. It stems from organizational culture—the unseen rules guiding organizational behavior. “Rules” should be interpreted as the means by which instructional designers will carry out the needs assessment, interact with the client, deliver results, interpret them, and plan action based on them. In developing the needs assessment plan, instructional designers should seek answers to such questions as these: • With whom in the organization should the instructional designer interact during the needs assessment? (How many people? For what issues?) • Whose approval is necessary to collect information? (For example, must the plant manager at each site grant approval for administering a questionnaire?)

34

Mastering the Instructional Design Process

• To whom should the results of the needs assessment be reported? To whom should periodic progress reports be provided, if desired at all? • How have previous consultants interacted with the organization? What did they do well, or what mistakes did they make, according to managers in the organization? • What methods of delivering results are likely to get the most serious consideration? (For instance, will a lengthy written report be read?) Instructional designers should always remember that the means by which needs assessment is carried out can influence the results and the willingness of the client to continue the relationship. It is important to use effective consulting skills appropriately. Assessing the Feasibility of the Needs Assessment Plan Before finalizing the needs assessment plan, instructional designers should review it with three important questions in mind: (1) Can it be done with the resources available? (2) Is it workable in the organizational culture? (3) Has all superfluous information been eliminated from the plan? It makes little sense to prepare an ambitious plan that cannot be carried out due to lack of resources, or accelerated project timelines with ridiculously short deadlines. Careful thought must be given to the resources. Instructional designers should ponder the following issues. • Given the draft needs assessment plan, what resources will be necessary to implement it successfully? • How many and what kind of people will be required to staff the effort? • What equipment and tools will they need? • How long will it take to conduct the needs assessment? • What limitations on staff, money, equipment, or access to information are likely to be faced, and is the needs assessment plan realistic in light of resources and likely constraints? Just as it makes little sense to establish an ambitious needs assessment plan that cannot be carried out with the resources available, it also makes little sense to plan a needs assessment not supported by the organizational culture. The following questions are also worth consideration. • How are decisions made in the organization, and how well does the needs assessment plan take the organization’s decision-making processes into account?

Conducting a Needs Assessment

35

• Whose opinions are most valued, and how well does the needs assessment plan consider their opinions? • How have organizational members solved problems in the past, and how well does the needs assessment plan take the organization’s past experience with problem solving into account? Finally, superfluous information should be eliminated from the needs assessment plan, needs assessment processes, and reports on the results. The acid test for useful information has to do with the persuasion that is necessary. Complex plans are unnecessary when decision makers require little information to be convinced of an instructional need. Too much information will only distract decision makers, drawing their attention away from what is important. Simplicity is more powerful and elegant.

Developing a Needs Assessment Plan: A Case Study Josephine Smith is the training director at a large Midwestern bank. She was recently hired for this job. As her first assignment, she was asked to review correspondence leaving the bank. Key officers of the bank have a problem of (in the words of one) “providing a tone in our correspondence that we put customer service first in whatever we do.” Josephine conducted an initial performance analysis (an alpha needs assessment) and found that the “problem” has several components. Each component she calls a “subproblem.” First, the bank uses form letters for most routine correspondence. Loan officers commonly send out these form letters, which were not written with an emphasis on a good “customer service tone.” This subproblem is a deficiency in the environment, and Josephine has asked the key officers to form a committee to review the letters and eventually revise them. They have agreed. Second, Josephine’s investigation reveals that employees at the bank do not know how to write correspondence with an adequate “customer service” tone. This sub-problem is a training need. Josephine set out to assess training needs by analyzing common problems appearing in nonroutine correspondence sent from the bank. She uses the results of this situation-specific, gamma-type needs assessment to identify the gap between what is (letters as written) and what should be (letters as they should be written). She will use that information in establishing instructional objectives for training that will furnish loan officers—her target audience—with the knowledge they need to write letters in desired ways.

36

Mastering the Instructional Design Process

Josephine begins needs assessment planning by proposing to her immediate superiors a review of special letters recently mailed from the bank by loan officers. These letters will be compared to criteria, set forth on a checklist, for letters exhibiting an adequate customer service tone. This checklist (an instrument) will be prepared by a committee comprising Josephine and several key managers in the bank. (The first step in developing the checklist will involve clarification of just what does and what does not constitute a good customer service tone, a phrase too vague to provide guidance in establishing concrete instructional objectives.) The same committee will then use the checklist to review letters and identify the frequency of common problems of tone in the letters. It will use the results to prioritize training objectives for loan officers.

Solving Problems in Conducting Needs Assessments Planning a needs assessment poses one challenge. Conducting the needs assessment—implementing the plan—poses another. While logic and research rigor are typically emphasized in the planning stage, everyday pressures to achieve quick results and hold down costs most keenly affect instructional designers during the implementation stage. However, implementation problems can usually be minimized if the plan has been stated and key decision makers have received advance notice of the plan and its pending implementation. The chances for success increase even more if key decision makers participate in developing the plan and feel ownership in it. When implementing the needs assessment plan, instructional designers should at least be able to apply tactics to ensure successful implementation. Tactics, perhaps best understood as approaches used in day-to-day operations, are necessary for dealing with common problems typically arising during implementation of a needs assessment plan. These problems include (a) managing sample selection, (b) collecting data while not creating false expectations, (c) avoiding errors in protocol, and (d) limiting participation in interpreting needs assessment results. Selecting a sample is usually simple enough. But contacting people or finding the “cases” selected is not always so simple. Sometimes people selected are not available because of absences from the job, pressures from work assignments and deadlines, or unwillingness to participate. “Cases”— such as documents or work samples—may be unavailable because they are being used for other reasons or are geographically beyond easy reach.

Conducting a Needs Assessment

37

The more employees who provide data about instructional needs, the higher people’s expectations will be that corrective action in the organization will take place. This expectation of change can be a positive force—an impetus for progressive change—when action quickly follows data collection and is visibly targeted on problems that many people believe should receive attention. However, the reverse is also true: the act of collecting data can be demoralizing when corrective action is delayed or when key managers appear to ignore the prevailing views of prospective learners about the direction for desired change. To overcome this problem, instructional designers can limit initial data collection efforts to small groups or to geographically restricted ones to hold down the number of people whose expectations are raised. Errors in protocol can also plague needs assessment efforts. Perhaps the most common one is the instructional designer’s failure to receive enough—or the right—permissions to collect data. To overcome this problem, instructional designers should discuss the organization’s formal (or informal) policies on data collection with key decision makers in the organization before sending out questionnaires, interviewing employees, or appearing in work units to observe job activities. They should make sure they have secured all necessary approvals before collecting data. Failure to take this step can create significant, and often unfortunate, barriers to cooperation in the organization. It may derail the entire needs assessment effort. Some instructional designers like to think of themselves as powerful change agents technically proficient in their craft and who, like skilled doctors, should “prescribe the right medicine to cure the ills” of the organization. Unfortunately, this approach is not always effective because it does not allow decision-makers to develop a sense of ownership in the solutions. They may think of the solution as “something dreamed up by those instructional designers.” To avoid this problem, instructional designers may form a committee of key managers to review the raw data and detailed results of their needs assessment before proposing a corrective action plan. Committee members go over the data and the analytical methods used. They are then asked for their interpretations and suggested solutions. This approach serves several useful purposes. First, it builds an informed constituency among the audience for the needs assessment report. Members of that constituency will grasp, perhaps better than most, how conclusions were arrived at. Second, they review raw data (occasionally, striking anecdotes or handwritten comments on questionnaires have a persuasive force that statistical results do not). Third, by giving members

38

Mastering the Instructional Design Process

of the committee an opportunity to interpret results on their own, instructional designers build support for the needs assessment’s results.

Identifying Instructional Problems Instructional designers should be capable of pinpointing instructional problems based on needs assessment results. The key to identifying instructional problems is the needs assessment plan itself. It should clarify what performance is desired and provide criteria by which to determine how well people are performing, how well people should perform, and how much difference there is between the two. By remembering what results are sought throughout the needs assessment process, instructional designers can prepare themselves for identifying instructional needs. One way to identify instructional needs is to focus, over the course of the needs assessment, on tentative needs that are discernible during the data collection process. To keep track, instructional designers may wish to use a needs assessment sheet. It is a structured way of recording instructional needs for subsequent review. Accountants use similar sheets when conducting financial, compliance, management, or program results audits. While the final results of the needs assessment may confirm these needs, the needs assessment sheets provide a means by which instructional designers can communicate with team members. They are also very helpful because they often suggest ways to organize the needs assessment report.

Ethical Issues in Needs Assessment A key ethical challenge in applying needs assessment can be expressed by this question: Can the needs assessment withstand charges it was not cost-effective, timely, or rigorous? Most instructional designers will find that their internal or external clients are not well-versed on what needs assessment is, why it should be conducted, how it should be conducted, or how long it should take. One manager has told an instructional designer that “If you have to take time to investigate the problem, then you are not aware of our business challenges and should seek employment elsewhere.” While that view may be wrongheaded, it underscores the need for instructional designers to educate their clients about what they do and to justify every step.

Conducting a Needs Assessment

39

To support a needs assessment, instructional designers should brief their clients on the instructional design process at the outset of their engagement. Needs assessment should be described for what it is—a way to economize efforts by targeting only the instruction that is necessary to solve or avert human performance problems. That will save time and money by avoiding investments in “sheep-dip training” that exposes all people to the same instruction despite unique individual or group needs. In a survey (Rothwell 2003), one respondent pointed to needs assessment and needs analysis as a frequent cause from which ethical challenges stemmed. The respondent bewailed that “it is often described as something that should be relentlessly and systematically done, but we are often pushed to action without the minimum analysis information.” There is a difference between what is contractually required and what is “right” (Hatcher 2002)—a major ethical dilemma encountered by those working in today’s frenetically paced organizations. Assessing Needs Cross-Culturally Needs assessment is prone to the same cross-cultural issues as performance analysis. Just as political climate can help or hinder performance analysis, so too can it help or hinder needs assessment. While no silver bullet exists to avoid all problems in all settings, a cultural informant should be identified and consulted before a needs assessment is conducted in a culture with which the instructional designer is unfamiliar. Cultural informants should be trustworthy and familiar with the local culture. In addition, they should understand why a needs assessment is important. Cultural informants should also be consulted about the language abilities of those targeted for participation in needs assessment and other cultural variables. For instance, if an instructional designer wants to administer a written needs assessment questionnaire, it will usually be necessary to determine in advance what language skills are possessed by those targeted to complete the questionnaire. It will also be necessary to determine whether the questionnaire should be written in English, the native tongue of the targeted participants, or both. (Translated questionnaires introduce a host of new requirements, too, such as the need to check translation accuracy.) Similarly, local customs—or the availability of technology—may also affect applications of other data collection methods. For instance, observation may prove distracting and troublesome to individuals in some cultures, so special steps may be necessary to make it

40

Mastering the Instructional Design Process

work as intended. If web-based questionnaires are administered, Internet speed may well be a consideration that will affect the outcome. The conclusion that the reader should draw from this chapter is that the entire process of needs assessment is undergoing a transformation that’s being driven by global factors affecting the instructional design field. These factors include (a) the need to integrate the entire training and development process more closely with the business, (b) increasing resource challenges (human and capital) that require substantial justification to grow or even maintain capability, (c) an increase in the demand to produce effective solutions in a shorter period of time, and (d) the role that advanced technology will continue to play in all aspects of the instructional design process, from needs assessment through evaluation. All these factors will continue to put pressure on today’s instructional designer to develop himself or herself professionally, not only in the instructional design field specifically, but in understanding the influences placed on business.

CHAPTER THREE

COLLECTING AND ANALYZING DATA FOR INSTRUCTIONAL DESIGN PROJECTS

B

efore an instructional designer or a performance improvement specialist can determine a course of action to address a perceived need, data must be gathered and analyzed. Some may argue this is the most critical part of the entire process, since what is discovered in the data collection and analysis phases will determine the direction that the solution or intervention should take. Whether the intention is to conduct an instructional needs analysis or a performance needs analysis, the intended outcome is the same: to identify the gaps (if any) that exist between where the target audience is now compared to where they could or should be. All of the data collection and analysis practices described in this chapter should achieve this outcome. The skill lies in knowing what data collection tools to use and how to analyze the results of the data collection process.

The Nature of Data Before deciding what data to collect and how it should be analyzed, we need to agree on a few basic concepts related to the nature of

41

42

Mastering the Instructional Design Process

data itself. Guerra-Lopez (2008) points out that data must meet three basic characteristics: 1. Relevancy: The data must directly relate to the research questions being answered. 2. Reliability: The data must be measured, trustworthy, and consistent. 3. Validity: The data must measure what we intend to measure (p. 135). Guerra-Lopez distinguishes between “hard” and “soft” data (an important consideration in our upcoming discussion of qualitative and quantitative measures). When we refer to “hard” data, we mean data that can be independently verified through external sources; that is, others studying the same situation using the same data collection and analysis process will reach the same conclusions. “Soft” data is represented by attitudes or inferences not independently verifiable of outside sources. Even the use of rating scales in the hope that the data can be considered “hard” (that is, numerically quantifiable) will not meet the standards of “hard” data. It is best to combine hard and soft measures to get a more complete picture. An Open or Closed Process? As has been pointed out earlier in this work, the instructional design process should be an “open” process; that is, it is cyclical, always requiring the instructional designer to review the status of the work and compare to previous stages, goals, or objectives. If need be, adjustments are made and the process continues to evolve based on constant feedback and revision. This phenomenon applies to the data collection and analysis phase. When viewed as a linear process, the steps are taken in order, beginning with a statement of purpose and a research question, then moving to the data collection and analysis phases and concluding with a summary of findings in a final report. When viewed as an open, or cyclical, process, however, several feedback loops might cause a readjustment. The Academy for Educational Development (2006) suggests that interpreting data may require a reinterpretation of the results of the data analysis methodology. In addition, once the data is disseminated to the organization’s stakeholders, the instructional designer may discover that the original research question was off target or not adequately framed. These iterative steps will strengthen the outcome of the data analysis process and improve future data collection and analysis procedures.

Collecting and Analyzing Data for Instructional Design Projects

43

One mistake that can easily be made is to choose one or more data collection tools without addressing the data needed to adequately answer the research question. The next section focuses on this issue.

Quantitative versus Qualitative Approaches Historically, quantitative research has been the foundation for the social sciences. Its strength is the reduction or elimination of bias, empirical testing, and the “wall of separation” between the researcher and his or her subjects. Qualitative research relies on a more constructivist approach; that is, the logic of the approach moves from the specific to the general in an environment that closely links the observer with the observed. As summarized by Terrell (2012), the quantitative tells us “if” and the qualitative tells us “how or why” (258). Terrell also points out the growing popularity of mixed methods (collecting both quantitative and qualitative data) in the fields of nursing, sociology, education, and other fields. By doing so, the researcher can paint a more complete picture of the environment under study. Debrovolny and Fuentes (2008) point out some significant differences between the qualitative and quantitative approaches that an instructional designer should consider before deciding on the data collection methods to be used in Table 3.1. Pershing (2006) suggests four myths that exist when comparing qualitative and quantitative methods. These are summarized in Table 3.2. In deciding which data collection tools to use, Debrovolny and Fuentes suggest that we answer preliminary questions whose answers will help us to determine the approach we should take. These questions include: • • • •

What research question are we trying to answer? Do we already have data we can use to answer our research question? What data do we need versus what data can we access? How much time have we been given to conduct the analysis (little time may guide us to quantitative methods, while more time might permit qualitative approaches or a combination of both)?

Let’s first turn our attention to describing the data collection tools available to us and how the data should be analyzed. The chapter will conclude with a comparative table showing the advantages and disadvantages of each.

44

Mastering the Instructional Design Process

TABLE 3.1. COMPARING QUANTITATIVE AND QUALITATIVE METHODS Quantitative

Qualitative

An assumption or hypothesis is made before data is collected

The analyst looks at the big picture, or context, and attempts to describe what is happening The analyst assumes that people see the situation differently from one another

The analyst assumes the situation is constant and everyone sees it the same way Requires a large random sample of people or data The observer is separated from the observed Results are described numerically Use predetermined theories to determine the data to be collected Analyze data using statistical data

The analyst selects specific data or people to study in greater depth The observer and the observed are known to each other and closely involved in the process Results are reported in words or stories Derive concepts and theories from the data that is collected Use inductive methods to analyze data, drawing patterns and inferences accordingly

Source: Adapted from Debrovolny and Fuentes (2008).

TABLE 3.2. QUANTITATIVE VERSUS QUALITATIVE METHODS: MYTHS AND REALITY Myth

Reality

The philosophical positions of quantitative and qualitative methods are incompatible

While both proceed from different premises, they produce more complete results when used in combination with one another Both approaches have standards of rigor that, when followed, will yield useful information. When not followed, both approaches will not yield useful outcomes A variety of approaches exist with both methods and expertise for both comes in selecting and combining the most appropriate tools for the situation This myth exists when, for example, a supposedly quantitative approach (e.g., a survey) in fact yields qualitative information (e.g., opinions)

Quantitative research is more rigorous than qualitative research

Quantitative research uses a variety of methodological approaches, while qualitative research methods are all the same Quantitative methods yield quantitative data while qualitative methods yield qualitative data Source: Adapted from Pershing (2006).

Collecting and Analyzing Data for Instructional Design Projects

45

The Data Collection Process There are a number of considerations when deciding which data to collect and how it should be collected. This section addresses these processes is detail. Establishing Sampling Procedures A sample is a small, representative group drawn from a larger group, called a population, and is used for quantitative data collection and analysis. Sampling is identifying smaller groups for examination. It is used to economize the time and expense of gathering information about needs, and it is often the focus of some questions (Thompson 2002). Any sample will deviate to some extent from the “true” nature of the population from which it is drawn, a principle known as sampling error. Sampling error cannot be eliminated, but it can be predicted and conclusions can be reached in a way that considers its effects. A sampling procedure is the method used to select a sample. Instructional designers commonly use any of four types of sampling procedures: (1) convenience or judgmental sampling, (2) simple random sampling, (3) stratified sampling, and (4) systematic sampling. To determine which one to use, instructional designers should consider the objectives of the needs assessment, certainty needed in the conclusions, the willingness of decision makers in the organization to allow information to be collected for the needs assessment study, and the resources (time, money, and staff) available. Convenience or judgmental sampling is probably used more often than many instructional designers would care to admit. It is a nonprobability sampling in that the subjects for review are chosen for convenience or accessibility rather than representativeness. Sampling is tempting because it is usually fast and inexpensive. Unfortunately, convenience or judgmental samples do not yield unbiased results because the choice of cases may be biased from the outset. To carry out convenience or judgmental sampling, instructional designers (1) select the number of cases to include in the sample based on convenience (they are easiest to obtain), access (capable of examination), or intuition (best guess of number to sample), and (2) choose the sample based on the results of Step 1.

46

Mastering the Instructional Design Process

Simple random sampling is probability sampling in which each subject in the population has an equal chance of being selected for study. This sampling procedure is appropriate when the population is large, and it does not matter which cases in the population are selected for examination. To carry out simple random sampling, instructional designers should (1) clarify the nature of the population, (2) list the population, (3) assign an identification number to each member of the population, and (4) select the sample by using any method that permits each member of the population an equal chance of being selected (use a random number table or the random number feature on certain electronic devices). Stratified sampling is more sophisticated. It is appropriate when the population is composed of subgroups differing in key respects. In needs assessment, subgroups may mean people in different job classes, hierarchical levels, structural parts of the organization, or geographical sites. They may also mean classifications of people by age group, level of educational attainment, previous job experience, or performance appraisal ratings. Stratified sampling ensures that each subgroup in a population is represented proportionally in a sample. For instance, suppose 10 percent of an organization comprises salespersons. If it is important in needs assessment to ensure that 10 percent of the sample comprises salespersons, then stratified sampling is appropriate. In simple random sampling, that may not occur. To carry out stratified random sampling, instructional designers should (1) clarify boundaries of the population, (2) identify subgroups within the population, (3) list members of each subgroup, (4) assign numbers to each member of each subgroup, (5) determine what percentage of the population comprises members of each subgroup, and (6) select the sample at random (each subgroup should be represented in proportion to its representation in the population). Systematic sampling is an alternative to other methods. It is simple to use. Suppose that it is necessary to assess the training needs of 10 percent of all employees in an organization. First make a list of everyone in the organization. Then divide the number of persons by 10 percent. Finally, select every tenth name on the list. If names are listed in random order, the resulting sample will be as good as a simple random sample. But if there is any order to the list whatsoever, the resulting sample may be biased because of that order. Many novices—and, occasionally, even those who are not novices— complain about sample size. On this subject, misconceptions are common. For instance, some people claim a sample size of 5 or 10 percent of a

Collecting and Analyzing Data for Instructional Design Projects

47

population is adequate for any purpose. Others may (jokingly) claim that any needs assessment is adequate if at least 345 cases are reviewed—because 345 is the minimum number of cases to achieve a representative sample of the entire U.S. population at a low confidence level! However, population size has nothing to do with sample size. Three issues should be considered when selecting sample size. First, consider degree of confidence. To be 100 percent certain, examine the entire population. But if lower degrees of confidence can be tolerated, the percentage of the population to be examined can be reduced. Second, consider maximum allowable error, indicating what number it may not exceed. Third, consider standard deviation. It measures variations in the population. When these numbers have been computed, appropriate sample size can be determined. Determining Data Collection Strategy and Tactics How will information about instructional needs be collected? Answer this question in the needs assessment plan, making sure that the data collection methods chosen are appropriate for investigating the performance problem. Five methods are typically used to collect information about instructional needs: (1) interviews, (2) direct observation of work, (3) indirect examination of performance or productivity measures, (4) questionnaires, and (5) task analysis. Other possible data collection approaches include (1) key informant or focus groups, (2) nominal group techniques, (3) Delphi procedure, (4) critical incident method, (5) root cause analysis, (6) competency assessment, (7) assessment center, and (8) exit interviews. To get a better picture of how these techniques work and where they should be used, let’s examine each. Interviews. Interviews are structured or unstructured conversations focusing on needs. Swanson (1994) points out that conducting effective interviews requires a good deal of skill and preparation. The author lists these skills:

• • • •

The ability to develop questions that will get meaningful answers. The ability to ask open-ended questions spontaneously. The ability to create an atmosphere of trust—not defensiveness. The ability to take complete and accurate notes without infusing one’s own ideas (p. 80).

48

Mastering the Instructional Design Process

Instructional designers should usually focus these conversations on key managers’ perceptions about the performance problem and the planned instruction to solve it. A key advantage of interviews is that they allow instructional designers the flexibility to question knowledgeable people, probing for information as necessary (Holstein and Gubrium 2001). A key disadvantage of interviews is that they may be time-consuming and expensive to carry out, especially if travel is required. To plan interviews, instructional designers should: • Prepare a list of general topics or questions. • Identify people knowledgeable about training needs. • Meet with the knowledgeable people and pose questions about training needs. • Take notes during or immediately following the interview. Direct Observations. Direct observations of work are, as the phrase implies,

first-hand examinations of what workers do to perform and how they do it. They may be planned or unplanned, they may rely on specialized forms to record the actions or results of performers, and they may even focus on behavior (Thompson, Felce, and Symons 1999). Indirect Observations. Indirect examinations of performance or productivity measures are called indirect because they are unobtrusive and do not require instructional designers to observe workers performing; rather, they judge performance from such tangible results or indicators of results as production records, quality control rejects, scrap rates, work samples, or other records about the quantity or quality of work performed. Indirect examinations may be structured (in which results of observations are recorded on checklists) or unstructured (in which the researcher’s feelings and perceptions about results are recorded). Questionnaires. Questionnaires, sometimes called mail or web-based

surveys, consist of written questions about instructional needs. They solicit opinions about needs from performers, their supervisors, or other stakeholders. They are sometimes developed from interview results to cross-check how many people share similar opinions or perceptions about needs (Dillman 1999). They may be structured (and use scaled responses) or unstructured (and use open-ended essay responses). In recent years, many people have moved from so-called paper-and-pencil questionnaires to web-based or web-supported questionnaires.

Collecting and Analyzing Data for Instructional Design Projects

49

Anecdotal evidence suggests that response rates as poor as 5 percent are not uncommon, and such low response rates are not helpful for drawing generalizations—although they may provide intriguing information for subsequent data collection efforts. Due to these shortcomings, a designer using a questionnaire to collect data should be prepared to: • Gain the support of the target group’s management so questionnaire recipients understand that their management is supporting the data collection process. • Keep the questionnaire to a reasonable length. Response rates are likely to be negatively affected if the instrument is perceived to be overwhelming. • Reassure the recipients (and their management) that the data will be held in strict confidence and that no one individual’s responses will be divulged. • Follow up with the responders after the data has been collected and analyzed so they have a general idea of what was discovered because of the process and how the information will be used. Task Analysis Task analysis is a general term for techniques by which work procedures or methods are carried out (Annett and Stanton 2001; Watson and Llorens 1997). While there are many variations to the task analysis process, it focuses on identifying the primary duties of a job, the tasks required to successfully fulfill each duty and the sub (or supporting) tasks required to complete each major task. A major challenge to the process is in creating a manageable set of tasks that can be validated and implemented. Many task analysis procedures take an additional step by labeling each major task based on three criteria: 1. Frequency: That is, how frequent is the task performed. If done infrequently, the instructional designer may need to develop supporting job aids for use when the task is required. 2. Difficulty: That is, how much skill, knowledge, or experience are needed to complete the task? A high level of difficulty implies extensive training on this task until proficiency is achieved. 3. Criticality: That is, how important is this task to the overall success of the job? If the task is not that critical, perhaps it should not be included in the overall task analysis or be dealt with in a more informal (rather than a formal training) way.

50

Mastering the Instructional Design Process

There are several additional data collection approaches that can be used effectively. These are summarized below. Key Informant Groups or Focus Groups. Key informant groups or focus

groups rely on highly knowledgeable people or committees composed of representatives from different segments of stakeholders (Bader and Rossi 2002; Krueger and Casey 2000). Key informant groups are especially knowledgeable about a performance problem or possible instructional needs; focus groups are committees, usually created informally, that pinpoints instructional needs through planned participation of representatives from key stakeholder groups. Nominal Group Technique. The nominal group technique (NGT) takes its

name from the formation of small groups in which the participants do not, during the earliest stages of data collection, actively interact. Hence, they are groups in name only—they are only nominal groups. To use NGT in data collection, instructional designers should: • Form a panel of people representative of the targeted learners (or their organizational superiors). • Call a meeting of the panel. • Ask each panel member to write opinions about training needs on slips of paper. • Permit no discussion as the opinions are being written. • Record items on a whiteboard or by using Post-It notes for subsequent panel discussion. • Combine similar responses. • Solicit discussion from panel members about what they have written. • Ask panel members to vote to accept or reject the opinions about training needs recorded earlier. The Delphi Procedure. The Delphi procedure takes its name from the

famed Delphic Oracle, well-known during ancient Greek times. Similar in some ways to NGT, the Delphi procedure substitutes written questionnaires for small-group interaction to collect information about training needs. To use the Delphi procedure to collect data, instructional designers should: • Form a panel of people representative of the target group. • Develop a written questionnaire based on the training needs or human performance problems to be investigated. Posing open-ended questions is acceptable at the outset. • Send copies of the questionnaire to panel members.

Collecting and Analyzing Data for Instructional Design Projects

51

• Compile results from the initial round of questionnaires and create scales to assess levels of agreement among the experts. • Prepare a second questionnaire and send it and the results of the first round to the panel members. • Compile results from the second round. • Continue the process of feedback and questionnaire preparation until opinions converge, usually after three rounds. The Critical Incident Method. The critical incident method takes its name

from collecting information about critically important (critical) performance in special situations (incidents). Critical incidents were first used as a method of collecting information about the training needs of pilots during World War II and were subsequently used to identify special training needs of CIA agents (Johnson 1983). To use the critical incident method, instructional designers should: • Identify experts such as experienced performers or their immediate supervisors. • Interview the experts about performance that is critical to success or failure in performing a job. • Ask the experts to relate anecdotes (stories) from their first-hand experience about situations in which performers are forced to make crucially important decisions. • Compare stories across the experts to identify common themes about what performers must know. • Use this information to identify training needs. Alternative approaches to this critical incident process may be used and may focus on the most difficult situations encountered, common daily work challenges, or the most common human performance problems observed with newcomers. Root Cause Analysis. The root cause analysis process was popularized by the

quality movement that drew a lot of attention in the 1980s. The strength of the methodology is that it not only attempts to answer the what and how questions, but also the why question. If we know why something happened, then we can get to the source of the problem and address the underlying causes so the issue will not occur again. There are several ways to depict the process, but, as Rooney and Vanden Heuvel (2004) point out, the process comprises four main steps: • Step One: Data collection. All information related to the problem is collected so it can be analyzed to uncover the eventual cause.

52

Mastering the Instructional Design Process

• Step Two: Causal factor charting. A chart is created that displays the sequence of events that led up to the problem occurrence and the conditions that surrounded the event. • Step Three: Root cause identification. Now that the potential causal factors have been identified, the investigators can now analyze the causes to identify the reason(s) for each cause. • Step Four: Recommendation generation and implementation. In this final step, the investigators present recommendations for preventing the problem from occurring again. Okes (2008) points out that effective root cause analysis can often be negatively influenced by human emotional characteristics. These include: • Recency bias. If the same or similar problem occurred recently, the current situation resulted from the same causes that affected the first problem. • Availability bias. In data collection, we tend to collect the data that’s easy to obtain rather than the data we should collect. • Anchoring bias. Latching onto the first piece of data we collect while ignoring other, more relevant data. • Confirmation bias. Collecting only evidence that supports our theory of what caused the problem rather than looking for evidence that might disprove our theory. Root cause analysis has proven to be an effective tool, when done properly, for uncovering the source of performance problems and correcting the causes to prevent future problems from arising. Competency Assessment. Competency assessment has been growing in

popularity in recent years (Rothwell and Graber 2010; Rothwell, Graber, Dubois, Zabellero, Haynes, Alkhalaf, and Sager 2015; Rothwell and Lindholm, 1999). Its purpose, according to one of many views, is to identify and isolate the characteristics of ideal (exemplary) performers (Dubois and Rothwell 2000). Those characteristics become a foundation for preparing instruction designed to raise average performers to ideal performers. A major advantage of competency assessment is that it is targeted toward achieving ideal performance more than rectifying individual performance problems or deficiencies. But a major disadvantage is

Collecting and Analyzing Data for Instructional Design Projects

53

that needs assessments using this form of data collection may be expensive and time-consuming to do if they are to be legally defensible. To use the competency assessment method, instructional designers should: • Form a panel of managers or experienced performers. • Identify the characteristics of ideal performers. (In this context, characteristics may mean behaviors, results achieved, or both.) • Pose the following questions to the panel members: What characteristics should be present in competent performers? How much should they be present? Answering these questions may involve behavioral events interviewing in which exemplary performers are asked to relate a significant work-related story from their experience and describe exactly what they did, how they felt as they did it, and even what they thought as they did it. • Devise ways to identify and measure the characteristics. • Compare characteristics of actual performers to those described in the competency model. • Identify differences that lend themselves to corrective action through planned instruction. Numerous alternatives to this approach exist. The reason: Views about what should be the basis for competencies may differ. According to one view, for instance, competencies are derived by studying the results (outputs) produced by performers; according to another view, competencies are derived from examining common characteristics shared by exemplary performers. Assessment Centers. An assessment center is not a place; rather, it is a

method of collecting information (Rothwell 2015; Thornton 1992). They are used to screen higher-level job applicants (those applying for senior leadership positions) or to assess the potential of existing employees for promotions. The process can range from a few assessment measures all the way to multiple measures and methods requiring days or even weeks to complete. Assessment centers are expensive to design and operate, which is a major disadvantage of this approach to data collection. However, their results are detailed, individualized, and job-related, and that is a chief

54

Mastering the Instructional Design Process

advantage of the assessment center method. To use the assessment center, instructional designers may have to rely on the skills of those who specialize in establishing them. The basic steps in preparing an assessment center are, however, simple enough. They require a highly skilled specialist, familiar with employee selection methods and testing validation, to: • Conduct an analysis of each job category to be assessed. • Identify important responsibilities for each job. • Use the results of Step 2 to develop games or simulations based on the knowledge and skills needed to perform the job successfully. • Train people to observe and judge the performance of participants in the assessment center. • Provide each individual who participates in the assessment center with specific feedback from observers about training needs. Performance Records. Marrelli (2005a and 2005b) points out that existing performance records can also serve as a way of informing critical incidents. Examples include exit interviews, performance appraisals, and work diaries. Exit interviews are planned or unplanned conversations carried out with an organization’s terminating employees to record their perceptions of employee training needs in their job categories or work groups. Exit interviews are relatively inexpensive to do and have high response rates. However, they may yield biased results because they highlight perceptions of employees who left the organization. Many instructional designers wonder when to choose one or more of these data collection methods. While there is no simple way to decide about choosing a method, several important issues identified by Newstrom and Lilyquist (1979, 56) in their classic treatment of this topic are still relevant:

• Incumbent involvement: How much does the data collection approach allow learners to participate in identifying needs? • Management involvement: How much does the data collection approach allow managers in the organization to participate in identifying needs? • Time required: How long will it take to collect and compile the data? • Cost: What will be the expense of using a data collection method? • Relevant quantifiable data: How much data will be produced? How useful will it be? How much will it lend itself to verifiable measurement? In considering various data collection methods, instructional designers are advised to weigh these issues carefully. Not all data collection methods share equal advantages and disadvantages.

Collecting and Analyzing Data for Instructional Design Projects

55

Work Samples. The outputs or actual products of work can be useful in the data collection and analysis phase because they represent the directly observable outcome of the work being studied that are collected. Because they represent the output of work, they have high validity, unlike the data collected using focus groups or interviews in which the information is subject to filtering and individual perceptions. Process Mapping. Process mapping is used to identify the steps taken in

sequence to produce a work output. The output of this process is a matrix or a process map. Such a map may also identify the time required to complete each step, a measure of success for each step, and the conditions under which the tasks are performed. The target group for process mapping is a representative group of performers or a small subset of top performers. Marrelli (2005b) points out that process mapping “ … can be an excellent approach to identifying the content that should be included in an instructional course, manual, or job aid intended to help workers execute a process” (41). Unobtrusive Measures. While our goal as instructional designers is to collect valid and relevant data we can use to make sound decisions, not every environment can be said to support this outcome. In some organizations, labor and management conspire, either knowingly or unknowingly, to produce the desired output through less-than-desirable behaviors. This concept is “shadowboxing with data,” in which work conditions or other impediments may be hidden by what appears to be desirable outcomes. One example that is much discussed today in education is the concept of “teaching to the test.” While scores may meet a standard, the behavior behind these results may be less than ideal. Winiecki (2009) describes the concept this way: “In shadowboxing with data, individuals who know what sorts of measured outputs are desired (by themselves or their organization) may modify their practice so as to produce ‘good numbers’ rather than what we consider to be ‘good performance’” (32). To address this phenomenon, we might consider unobtrusive measures described as follows. Marrelli (2007) points out that many of the commonly used data collection methods (surveys, focus groups, questionnaires, interviews) are inherently biased by their altering the situation being studied, they are reactive and the participants, by agreeing to participate, will naturally impose their own biases and perceptions. These limitations make it “ … difficult to distinguish between typical, real behaviors and behaviors induced by the measurement” (44).

56

Mastering the Instructional Design Process

To compensate for these limitations, unobtrusive measures correct for this limitation. They can be grouped into three categories: (1) physical traces; (2) archives; and (3) observations. An education program that encourages recycling of used paper can be assessed by the volume of recycled paper before and after the program (physical traces). Data stored or collected by an organization not intended for analysis can be a rich source of information (archives). Target groups performing a job or task under analysis can be observed without their knowledge (observations) to gather valid and reliable information. Specifying Instruments and Protocols. What instruments should be used

during the needs assessment, and how should they be used? What approvals or protocols are necessary for conducting the needs assessment, and how will the instructional designer interact with members of the organization? These questions must be addressed in a needs assessment plan. The first has to do with specifying instruments; the second has to do with specifying protocol. Many instruments may be used in needs assessment. Common methods of collecting information about instructional needs rely on commercially available or tailor-made questionnaires, interview guides, observation guides, tests, and document review guides. Commercially available instruments and online data collection methods have been prepared for widespread applications, although some consideration of how to use an instrument or groupware program in one organizational setting is usually necessary and should be described in the needs assessment. Tailor-made instruments are prepared by instructional designers or others for assessing instructional needs in one organization or one job classification. Developing a valid, reliable questionnaire may require substantial work in its own right, and this process should be described in the needs assessment plan. Using groupware necessitates establishing an approach to data collection. Table 3.3 summarizes methods of needs assessment. Protocol means diplomatic etiquette and must be considered in planning needs assessment. It stems from organizational culture—the unseen rules guiding organizational behavior. “Rules” should be interpreted as the means by which instructional designers will carry out the needs assessment, interact with the client, deliver results, interpret them, and plan action based on them. In developing the needs assessment plan, instructional designers should seek answers to such questions as these: • With whom in the organization should the instructional designer interact during the needs assessment? (How many people? For what issues?)

Collecting and Analyzing Data for Instructional Design Projects

57

TABLE 3.3. STRENGTHS AND WEAKNESSES OF SELECTED DATA COLLECTION METHODS Criteria

Methods Interviews Direct observation of work Indirect examinations of performance or productivity measures Questionnaires Task analysis Key informant or focus groups Nominal group technique Delphi procedure Critical incident method Competency assessment Assessment center Exit interviews

Incumbent Management Time Involvement Involvement Required Cost

Relevant Quantifiable Data

High Moderate

Low Low

High High

High High

Moderate Moderate

Low

Moderate

Low

Low

High

High Low High

High Low Moderate

Moderate Moderate High High High High Moderate Moderate Moderate

High

Moderate

Moderate Moderate Moderate

Low

Moderate

Moderate Moderate Moderate

Moderate

Moderate

Low

Low

Low

Low

High

High

High

High

High

Low

High

High

High

Low

Low

Low

Low

Low

Source: J. Newstrom and J. Lilyquist. Reprinted from Training and Development Journal (1979), 56. Copyright © 1979. The American Society for Training and Development. Reprinted with permission. All rights reserved.

• Whose approval is necessary to collect information? (For example, must the plant manager at each site grant approval for administering a questionnaire?) • To whom should the results of the needs assessment be reported? To whom should periodic progress reports be provided, if desired at all? • How have previous consultants interacted with the organization? What did they do well, or what mistakes did they make, according to managers in the organization?

58

Mastering the Instructional Design Process

• What methods of delivering results are likely to get the most serious consideration? (For instance, will a lengthy written report be read?) Instructional designers should always remember that the means by which needs assessment is carried out can influence the results and the willingness of the client to continue the relationship. Use effective interpersonal skills.

Data Analysis Before discussing the data analysis process, we should know of the answers to several critical questions that must be addressed early in the entire data collection and analysis process. How will results of the needs assessment be analyzed once the information has been collected? This question must be answered in a needs assessment plan. It is also the one question that instructional designers may inadvertently forget. But if it is not considered, then subsequent analysis will be difficult because instructional designers may find they did not collect enough information, or they collected the wrong kind to make informed decisions about instructional needs. Selecting a data analysis method depends on the needs assessment design, corresponding to a research design previously selected. They include: (1) historical, (2) descriptive, (3) developmental, (4) case or field study, (5) correlational, (6) causal-comparative, (7) true experimental, (8) quasi-experimental, and (9) action research (Isaac and Michael 1984). Historical and case or field study designs usually rely heavily on qualitative approaches to data analysis. The instructional designer describes conditions in the past (historical studies) or present (case or field study). Hence, analysis is expressed in narrative form, often involving anecdotes or literature reviews. Anecdotes have strong persuasive appeal, and they are selected for their exceptional or unusual nature. They are rarely intended to represent typical conditions or situations. Descriptive designs include interview studies, questionnaires, and document reviews. Data are presented either qualitatively as narrative or quantitatively through simple frequencies, means, modes, and medians. A frequency is little more than a count of how often a problem occurs or an event happens. A mean is the arithmetic average of numbers. A mode is the most common number, and the median is the middle number

Collecting and Analyzing Data for Instructional Design Projects

59

in a sequence. Perhaps examples will help to clarify these terms. Suppose we have a series of numbers: 1, 4, 9, 7, 6, 3, 4. The frequency is the number of times each number occurs. Each number occurs one time, except for 4. The mode of this series of numbers then is 4, since it occurs most frequently. The median is the middle number, found by arranging the numbers in order and then counting: 1, 3, 4, 4, 6, 7, 9. The median in this array is 4, since it is the middle number. To find the mean (arithmetic average), add the numbers and then divide by how many numbers there are. Here, the sum of 1 + 4 + 9 + 7 + 6 + 3 + 4 equals 34 divided by 7 equals 4.8 (rounded). Frequencies, means, modes, and medians are used in analyzing needs assessment data because they are simple to understand and are also simple to explain to decision makers. In addition, they lend themselves especially well to the preparation of computerized graphics. The analysis used in other needs assessment designs—developmental, correlational, experimental, quasi-experimental, or causal-comparative— requires sophisticated statistical techniques. For these designs, the most commonly used data analytical methods include the analysis of variance, chi square, and the t test. When these methods must be used, instructional designers should refer to detailed descriptions about them in statistics textbooks. Before selecting the data collection tools used, the instructional designer must decide whether quantitative data, qualitative data or a combination of both must be gathered to answer the needs assessment question(s). Whether we’re involved in assessing learning or performance gaps, the data collection and analysis phase of the process is critical to the outcome. If we don’t know the data to collect or how to analyze it, the outcome of our efforts will be less than stellar. From a high level, the instructional designer must answer three basic questions in this phase: 1. What are the needs this project is attempting to satisfy? 2. What data or information will I need to clarify these needs? 3. How will I need to analyze this data to arrive at a reliable and valid conclusion? The answers to these three questions should be front and center in our needs assessment plan. They should be communicated to our clients

60

Mastering the Instructional Design Process

and rigorously upheld during the process. As members of the instructional design community, we are ethically bound during the data collection and analysis phase to: • Avoid “confirmation bias”; that is, identifying a predetermined outcome at the beginning, then collecting and analyzing only the data that supports our predetermined conclusion. Establish guidelines at the beginning with the client and stakeholders about what data will be collected and how it will be used. • Select the most appropriate methods based on the general data needed to address the needs assessment questions (that is, quantitative, qualitative, or a combination of both). • Avoid any attempt to manipulate or otherwise hide data for political or other reasons. Now that we’ve reviewed the steps required to complete the data collection and analysis phases of our project, we are ready to move on to the needs assessment phase itself in the following chapter.

CHAPTER FOUR

IDENTIFYING THE TARGET POPULATION AND ENVIRONMENTAL CHARACTERISTICS

L

earners are not all alike. Members of different occupations and individuals differ in the ways they learn best. As instruction is prepared, these differences must be considered. To do that successfully, instructional designers should know the characteristics of the targeted learners. Identifying these characteristics is called assessing relevant characteristics of learners, although we will call it learner assessment. As we’ve noted earlier, the role of an instructional designer today has expanded to include a broader consideration of performance; that is, not only instructional solutions but a myriad number of performance improvement solutions that have little to do with skill or knowledge deficiencies. With this as the case, learner characteristics must also consider the work environment and the culture in which the performer operates. Other modern-day considerations include a much more diverse workforce than in the past, reflected by a multicultural and cross-generational workforce. These issues will be addressed in this chapter. According to The Standards, the performance standards related to identifying and describing the target population and environmental characteristics include: “(a) determine the characteristics of the target population that may impact the design and delivery of instruction (essential); (b) determine characteristics of the physical, social, political, and cultural environment that may influence learning, attitudes, and performance 61

62

Mastering the Instructional Design Process

(advanced); (c) identify the infrastructure that is available to support the design and delivery of instruction (advanced); (d) determine the extent to which the organizational mission, philosophy, and values may impact the design and delivery of instruction (advanced); and (e) analyze, evaluate, and use learner profile data and environmental characteristics to design instruction (essential)” (Koszalka, Russ-Eft, and Reiser 2013, 40–41). In this chapter, we describe the selection of learner characteristics for assessment, the identification of ways of determining the physical, social, political, and cultural characteristics that affect learning, the identification of the infrastructure available to support the design and delivery of instruction, the impact of organizational mission, philosophy, and values on the design and delivery of instruction and, finally, using learner profile data and environmental characteristics to design instruction.

Selecting Learner Characteristics for Assessments Before preparing instructional or training materials, instructional designers should be able to answer this simple question: Who is the intended learner? The answer helps define the target population, target group, or target audience. Traditionally, writers on this subject have advised instructional designers to direct their attention to typical or representative learners to maximize the number (and success rates) of people who subsequently participate in instruction (Blank 1982). However, growing sensitivity to the needs of atypical learners, such as those possessing physical, mental, or learning disabilities, may require instructional designers to pay increasing attention to a broader range of learner characteristics. According to Cornell University’s 2012 Disability Status Report, 10.4 percent of working-age adults (21–64) have a disability. Employers are also expected to provide reasonable accommodations for learning disabilities under the Americans with Disabilities Act discussed later in this chapter. What Learner Characteristics Should Be Assessed? Assessing learner characteristics resembles segmentation, the process used to categorize consumers by similar features. A well-known technique in the advertising and marketing fields, segmentation gives advertisers the ability to target messages to the unique needs and concerns of their audiences. In similar fashion, learners are consumers of services provided by instructional designers. Many fundamental marketing principles apply to

Identifying the Target Population and Environmental Characteristics

63

assessing learner characteristics. Much like how an organization competes against other organizations in the marketplace, instruction must compete with other priorities for the attention of learners and their supervisors. The key to success in instructional design is to produce a product that is a perfect fit for the intended audience. When the audience sees the benefits of the solution and accepts it as a valuable resource, success is assured. Three basic categories of learner characteristics relate to a situation, performance problem, or instructional need: situation-related characteristics, decision-related characteristics, and learner-related characteristics. Situation-Related Characteristics. Situation-related characteristics stem

from events surrounding the decision to design and deliver instruction. The chief focus of the instructional design effort should be directed to those most affected by it. The reason: subsequent delivery of instruction to that group will presumably have the greatest impact. It will also be substantially more cost-effective than delivering instruction to all employees when only some really need it. Suppose that customers of one organization complain that they are not being treated courteously over the phone. Performance analysis reveals that it is a problem caused by a lack of knowledge of phone courtesy. In this simple example, the performance problem itself suggests an important learner characteristic: training should be designed only for those using telephones and dealing directly with customers. Since not all employees in an organization use phones or deal with customers, this learner characteristic alone is helpful in narrowing down the target audience. It raises additional questions. For instance, what do these employees have in common that (perhaps) others do not? Why do they talk to customers? When assessing situational characteristics of learners, instructional designers should begin by asking this question: What are the possible relationships between the performance problem and the learner? Does the performance problem itself suggest unique characteristics of the learners who should receive instruction? If so, what are they? Will those characteristics remain the same—or change—over time? If they will change, in what ways will they change? Decision-Related Characteristics. Decision-related characteristics pertain to

those deciding about learner participation in instruction. When assessing these characteristics, instructional designers should ask the following question: Who decides about permitting people to participate in instruction? Instructional designers may prepare instruction for a targeted group, but

64

Mastering the Instructional Design Process

others often decide who participates. If this is ignored, much time may be wasted preparing instruction to meet the needs of one group, only to find that other groups actually participate. Instructional designers should clarify, before preparing training materials, who will decide about participation. There are several ways to do that. One way is to establish a formal committee of people from inside the organization. The members of the committee can give advice about who should participate, predict who is likely to participate, and offer practical guidance for attracting participants by targeting the needs of decision makers. Learner-Related Characteristics. Learner-related characteristics stem from learners themselves. There are two kinds: (1) prerequisite knowledge, skills, and attitudes; and (2) other learner-related characteristics. Prerequisite knowledge, skills, or attitudes is sometimes called simply a prerequisite. In a classic treatment, Blank (1982, 44–45) defines a prerequisite as “a characteristic, trait, or ability that students should possess to be successful on the job—but one that they will not get as a result of a training program.” Blank identifies four types: (1) physical traits, (2) previously learned skills, (3) previously learned knowledge, and (4) previously learned attitudes. Physical traits include manual dexterity, grip strength, lifting ability, visual acuity, hearing ability, tolerance to extreme conditions, height, weight, sense of balance, and sensitivity to chemicals or other substances. Employers must take care to make reasonable accommodation, too, for workers with physical and other disabilities who can perform various jobs, but perhaps with modifications. Previously learned skills include the ability to read, write, and compute at a certain minimum level, the ability to use certain types of machines or tools, the ability to drive vehicles (forklift, road grader, tractor), and keyboarding skills. Previously learned knowledge includes awareness of rules such as those associated with arithmetic, grammar, pronunciation, electricity, chemistry, or medicine. Previously learned attitudes include basic employability skills, such as awareness of the importance of dress, punctuality, interpersonal relations at work, and organizational policies and procedures (Carnevale, Gainer, and Meltzer 1988). There is no foolproof method for establishing instructional prerequisites; rather, it is often a trial-and-error process. Often, instructional designers must ask themselves what knowledge, skills, and attitudes they think participants will bring with them to instruction. Later, when instructional materials and methods are tested on small groups of learners chosen as representative of the targeted audience, assumptions made about prerequisite

Identifying the Target Population and Environmental Characteristics

65

knowledge, skills, and attitudes can also be tested. Another approach is to select at random a few prospective participants to see if they possess the prerequisites. Instructional designers should remember two key points as they identify prerequisites. First, if trainees enter instruction lacking essential knowledge or skills, then these essentials must be furnished to them. Second, competent legal advice should be sought before people are screened out of instruction for job advancement or security, particularly when physical traits are the prerequisites. The reason is that using physical requirements in screening, while superficially appearing to have a neutral effect on the selection of protected labor groups, may screen out higher proportions of females and others. When instruction is necessary for job entry or advancement and is denied to some individuals solely because they do not meet previously established prerequisites about physical ability, then it functions as a selection device. Instruction is subject to the laws, regulations, and court decisions affecting equal employment opportunity and equal access for the disabled. Other learner-related characteristics are also worthy of consideration. They center on the learners’ demographic characteristics, physiological characteristics, aptitudes, experience, learning styles, attitudes, job categories, value systems, life-cycle stages, or career stages. The following summarizes the terms designating these important characteristics. Demographic characteristics include age, gender, and race; physiological characteristics include heart condition, lung capacity, and general physical condition. Experience characteristics include length of service with the organization, length of service in the job, experience with present job activities prior to job entry, and similar experience; learning style characteristics are classified according to standardized categories. Aptitude includes talents and skills; knowledge includes education, basic skills, and specialized previous training. Attitudinal feelings include feelings about the topic, training, the job, performance problems, and the organization. A more in-depth look at these terms follows. Demographic characteristics are associated with learners’ race, gender, and age. Two demographic issues are worthy of special consideration. First, instructional designers should ask whether the instruction they design will be geared to the needs of a particular racial group, gender, or age group, as is sometimes the case in specialized seminars on career planning, communication, retirement, or other subjects. If it will be, then any assumptions made about the learners should be double-checked.

66

Mastering the Instructional Design Process

These assumptions may be based, knowingly or unknowingly, on stereotypes about the needs or beliefs of the targeted audience and may be erroneous. To avoid this problem, some instructional designers may establish an ad hoc panel of advisers to clarify or check the assumptions made about the learners before instruction is designed. Much interest lately is focusing around the needs of older learners, people over the traditional retirement age, since Baby Boomers are aging and may soon become a central focus of interest to many employers strapped for talent (Charness and Czaja 2006; Moseley and Dessinger 2007; Rothwell, Sterns, Spokus, and Reaser 2008). Learner sensitivity to special issues is a second matter to be considered. In recent years, for instance, much attention has been devoted to establishing gender-neutral language to avoid stereotypes or other objectionable implications about the gender of employees. Are there other issues in the instruction that must be considered from the standpoint of unique employee groups? If so, they should be identified. Further, means should be established, before instruction is designed, to make sure that learner sensitivities are not violated and that human diversity is celebrated. Physiological characteristics pertain to the most intimate aspects of the learner. They may include sensitivity to chemicals, prior medical history, and genetic heritage, including a tendency to certain forms of disease. As medical science has advanced, it has become more than a science fiction writer’s dream to assess—and even predict—human sensitivity to substances and inclinations to disease. Relatively little attention in the literature has been devoted to making assumptions about learners’ physiological conditions. If learners will be exposed to chemicals during instruction, then their physiological characteristics should be considered. Should they be given medical examinations before exposure? Have all government requirements been met so that employees know of their “right to know” about the substances to which they will be exposed? Aptitudes are the future capabilities to perform in certain ways. Some individuals are gifted with talents that others do not possess, and those talents are synonymous with aptitudes. Employers sometimes administer aptitude tests before or after employee selection to assess individual potential. When test scores are available, they can be a rich source of information about learners. While this information may be used in designing instruction, it should be examined with due consideration to organizational policies on employee confidentiality and rights to privacy.

Identifying the Target Population and Environmental Characteristics

67

Experience means the time the targeted learners have spent in the employing organization, in their jobs, and in their chosen occupations. It is frequently one of the most important learner characteristics to consider in designing instruction. There are several reasons. First, experience sometimes affects motivation to learn. When people first enter an organization, job, or occupation, they are often highly motivated to learn. They want to reduce the tension existing between themselves and the unfamiliar surroundings (organization) or unfamiliar activities and expectations they face. When instruction will be designed for those with limited experience, there is a greater likelihood that the targeted learners will be motivated to learn. These learners are willing to take instruction seriously and may depend on it to help them make essential transitions in their lives. Second, experience affects the selection of appropriate instructional methods. Learners with the least experience need the most guidance. Since they “do not know what they do not know,” they are prime candidates for directive methods or simulated experiences. However, learners with the most experience rebel against directive instructional methods or unrealistic simulations. Knowledge is associated with what learners know about the subject of instruction, the performance problem, learning needs, and organizational policies and procedures. What is known about the learners’ knowledge of these subjects? What assumptions are safe to make about what they know before they enter instruction? Have learners had much or little formal education? Have they had specific, previous instruction on the subject at another institution? If so, how was the subject treated? Learning styles have been another learner consideration for years. Inventories have been developed around some of the major learning style paradigms. Recently, however, evidence-based research has called this learner characteristic into question. Clark (2010) is quoted as saying that “learning style represents one of the most wasteful and misleading pervasive learning myths in the past 20 years” (p. 10). The style most in question appear to be the approaches based on auditory, visual, and kinesthetic learning styles. Recent research indicates that we all learn using all three sensory modalities. Attitudinal characteristics refer to learners’ feelings about performance that they voice to other people. The term specifically denotes what learners think about a subject, the performance problem that instruction solves, their own learning needs, the organization, and other important issues. One way instructional designers can assess attitudes is to prepare and administer a simple attitude survey to representatives

68

Mastering the Instructional Design Process

of the targeted audience. Another way is to field-test instructional materials and then administer an attitude survey to participants in a small-group session. Geographical location may affect learners’ needs and willingness to participate in instruction. It may also influence their attitudes about the performance problem and the instruction designed to address it. Learners in different parts of the world may report to different supervisors and may face problems differing in degree or type from learners in other locations. Marketing specialists stress the importance of geographical dimensions as a basis for segmenting markets. Instructional designers may wish to target instruction to one geographical area first and then, in time, to spread out to others. This method is frequently used in marketing products or services. Job category means the learners’ job duties and responsibilities within the organization. It can be an important determinant of what employees must know and do to perform satisfactorily. Job categories often become the basis for establishing long-term instructional plans to make it easier to orient people to new jobs, upgrade their knowledge and skills as job requirements change, and prepare individuals for promotion or other movements. In some organizations, team or individual assignments—or some other method—may be substituted for job category if they are the primary means by which work is organized. There is good reason for placing heavy emphasis on job or work categories when assessing learner characteristics. The work performed is a key link between individual and organizational needs. Individual needs and characteristics also vary somewhat by job or by work responsibilities. Hourly employees may not need the same instruction as supervisors, managers, or executives on a given organizational policy or procedure. Instruction targeted for one employee category should take the duties and responsibilities of that category into account. However, jobs may be categorized in several ways. For instance, some instructional designers prefer to use a general job classification scheme. Examples of general job categories include executives, managers, first-line supervisors, technical employees, salespersons, professionals, and skilled workers. An alternative classification scheme, established by the Equal Employment Opportunity Commission for mandatory government reports on hiring, training, and other employee activities, lists the following job categories: officers and managers, professionals, technicians, sales workers, office and clerical workers, skilled craft workers, semiskilled operatives, unskilled laborers, and service workers. The actual job titles placed in each job category may vary across organizations but should remain consistent within one organization.

Identifying the Target Population and Environmental Characteristics

69

Barbazette (2006) cites additional learner characteristics that might also be included in the learner analysis process: • Interests. Knowledge of what the learners find interesting might lead to the development of adaptation of exercises or games that the learners would find relevant and engaging. • Personal benefit to learning. This relates to the WIIFM (“What’s in it for me?”) that can serve as a motivational factors if we know what the learner would find to be beneficial. • Cultural characteristics. The more we know about the culture that might influence the learners, the better able we are to adapt the instructional design to their needs. This not only means national culture or language restrictions but also corporate culture or the culture of a particular job or work setting. Physical, Social, Political, and Cultural Influences Beyond the characteristics of the learner or the learner group, the designer must consider the environment in which the performer works. Value systems are, according to one classic definition, “enduring organizations of beliefs concerning preferable modes of conduct or end-states of existence along a continuum of relative importance” (Rokeach 1973, p. 5). They are closely associated with organizational culture, perhaps best understood as the taken-for-granted assumptions about the “right” and “wrong” ways of behaving and performing in a particular setting (Schein 1985). To be effective, instruction should be designed with multiple value systems considered. Rokeach included a questionnaire in his classic book, The Nature of Human Values (1973), which remains very useful in assessing the value systems of people in organizational settings. Instructional designers may administer this survey before instruction is designed or before it is delivered to a group in one instructional session. Another classic book, Unblocking Organizational Values by Francis and Woodcock (1990), also provides information for assessing individual values. Life-cycle stage pertains to the individual’s age and stage of development. In each stage of development, the individual experiences central life crises that stimulate interest in learning about issues related to those crises. The life-cycle stages of prospective participants in instruction are worth some consideration by instructional designers. The crucial importance of life cycles was first recognized by the developmental psychologist Erikson (1959). It has since been popularized by Levinson (1978) and Sheehy (1974). The importance of life-cycle stage in designing instruction was first recognized by Havighurst (1970), described

70

Mastering the Instructional Design Process

more completely by Knox (1977), and reinforced by Knowles (1984) and Knowles, Swanson, and Holton (2005). Knowles, for instance, identifies three specific stages of adulthood and describes typical “life problems” associated with them, based on vocation or career and home and family living. The three stages are early adulthood (age 18 to 30), middle adulthood (age 30 to 65), and later adulthood (age 65 and over). During early adulthood, as Knowles points out, most people are exploring career options, choosing a career, getting a job, learning job skills, and making career progress. They are also usually dating, selecting a mate, preparing for marriage and family, and accepting many responsibilities of adulthood, such as purchasing a home, raising children, and making repairs. They are primarily interested in learning about improving their employment-related skills, clarifying their personal values, and coping with the responsibilities of the first stage of adulthood. During middle adulthood, most people face somewhat different life problems. They learn advanced job skills and move beyond technical and into supervisory work. They cope with the challenges of teenage children, adjust to aging parents, and plan for retirement. They are chiefly interested in self-renewal and in dealing with change. During later adulthood, most people encounter challenges very different from those of the middle years. They must adjust to retirement. They may have to adjust to the death of a spouse or learn how to deal with grandchildren. Their central learning issues have to do with keeping up-to-date and coping with retirement. Career stages or career prospects may also influence learners. Several career theorists have suggested that individuals progress through identifiable career stages (for example, Dalton, Thompson, and Price 1977). Examples of such stages include apprentice, colleague, mentor, and sponsor. These stages and their potential influence on instruction are described in Table 4.1. Recent attention has focused on integrating career planning and succession management, and that attention can have implications for what development programs are needed to build competencies (Rothwell, Jackson, Ressler, and Jones 2015). Consider the career stages of the targeted learners. Learners who view instruction as a vehicle for career advancement—as those in the apprentice stage are likely to do—will want practical, hands-on instruction that can help them advance in their careers. Other learners will not. They will see instruction as serving other purposes. All of these factors, regardless of which descriptive model you follow, suggests that we need to understand the people we train and how their perspectives might affect their learning potential.

Identifying the Target Population and Environmental Characteristics

71

TABLE 4.1. SUMMARY OF STAGES IN THE CLASSIC DALTON, THOMPSON, AND PRICE MODEL Stage

Focus

Affects Instruction

Apprentice

Performs technical work Deals with authority Learns from others about work and about dealing with others Begins to specialize Regarded as competent Makes contacts Provides leadership Develops more contacts Demonstrates ability to get things done Initiates programs Guides others Continues to develop contacts

Interest in techniques and technical issues Interest in dealing with others

Colleague

Mentor

Sponsor

Interest in maintaining professional competence Interest in dealing with others Interest in guiding/influencing others Interest in exerting long-term impact by influencing “up-and-coming” people

Source: W. Rothwell and H. Kazanas. Human Resource Development: A Strategic Approach, Copyright © 2004, p. 362. Reprinted by permission of HRD Press, Amherst, MA.

Selecting Learner Characteristics: A Case Study Example Georgeanna Lorch is an instructional designer hired as an external consultant to design and implement a new management performance appraisal system for Ajax Vending Company, a wholly owned subsidiary of a much larger corporation. The new appraisal system will be used with all supervisors, managers, executives, professionals, and technical workers at Ajax. As part of her contract, Lorch is preparing instruction on appraisal for managers and executives in the company. Lorch assesses learner characteristics by brainstorming and by completing the Worksheet on Learner Characteristics appearing in Exhibit 4.1. When she has completed the worksheet, she has identified most of the crucial learner characteristics that will affect her project. Later, she discusses—and double-checks—the learner characteristics with members of the organization and randomly selected representatives of the targeted audience.

72

Mastering the Instructional Design Process

EXHIBIT 4.1. A WORKSHEET ON LEARNER CHARACTERISTICS Directions: Use this worksheet to help you structure your thinking on learner characteristics that may—or should—influence your instructional design project. For each learner characteristic listed in column 1 below, identify in column 2 what learner characteristics are unique to the situation. Then, in column 3, describe how the characteristic(s) should be addressed or considered in the instruction that you subsequently design.

Column 1

What learner characteristics …

Column 2

Column 3

What are the characteristics?

How should the characteristics be addressed (or considered) in the instruction you subsequently design?

Are targeted directly at the area of need? Pertain to organizational policies? Pertain to learner/organizational needs? Can be addressed with available resources? Pertain to existing constraints on the instructional design project? Are feasible to collect data about in terms of resources and logistical limitations? Are translatable into design specifications? Are related to the performance problem that instruction is intended to solve? Other: Source: Mastering the Instructional Design Process: A Systematic Approach (4th ed.). Copyright © 2008 by John Wiley & Sons. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www .pfeiffer.com

Identifying the Target Population and Environmental Characteristics

73

When Should Learner Characteristics Be Assessed? Learner characteristics should be assessed at three points in the instructional design process. First, instructional designers should consider the targeted learners before instruction is prepared to meet identified instructional needs and solve human performance problems as they exist. As they do that, they should clarify exactly what assumptions they make about the knowledge, skills, and attitudes typical of intended learners. Instruction should be designed but it should be clarified how prospective participants may satisfy necessary prerequisites through means other than instruction. These assumptions can be tested later during formative evaluation of the instruction. Second, instructional designers should consider targeted learners who may need to participate in future instruction, perhaps regularly. These learners will be affected by the selection and promotion practices of the organization, which will determine the appropriate entry-level knowledge, skills, and attitudes of people moving into the jobs. Future learners may have needs—and the organization may experience human performance problems—uniquely different from those existing when instruction is first designed or delivered. For instance, job duties may change. Likewise, the organization may shift strategic direction and change performance requirements of every position. Then, too, new technology and work methods may be introduced. These changes (others can be identified as well) may dramatically affect the learner characteristics to be considered. Hence, instructional designers should forecast learner characteristics that may need to be considered for designing effective instruction (Rothwell and Kazanas 2003). Third, instructional designers should consider characteristics of a specific targeted group of learners each time the instruction is delivered (Knowles 1980). One group or one individual may have a unique profile, perhaps one different from the typical or representative characteristics of most learners in the organization. If radical differences between an actual targeted group of learners and the average or typical learners are ignored, major problems will be experienced during delivery. How Should Learner Characteristics Be Assessed? Instructional designers may assess learner characteristics using either of two methods: the derived approach or the contrived approach.

74

Mastering the Instructional Design Process

The derived approach is simplest to use. Can instructional designers identify learner characteristics of obvious importance to a performance problem, instructional need, or organizational constraint by brainstorming? If so, they can derive learner characteristics. If relevant learner characteristics can be identified in this way, then a list of learner characteristics to consider during instructional design will usually suffice. The process can be simple. However, the contrived approach may not be as simple to use. If learner characteristics cannot be identified easily through the derived approach, then the instructional designers should contrive a list of characteristics worthy of consideration. They should then go through the general list item by item, asking themselves whether each item is related to the performance problem to be solved, the instructional needs to be met, or the organization’s policies and procedures. Unrelated items on the list can be ignored; related items must be pinpointed. Expensive and time-consuming methods of assessing learner characteristics are unnecessary usually. Often, instructional designers and line managers already know of the people for whom instruction is being designed. All that is necessary is to write out that profile of the prospective learner and verify its accuracy with others such as line managers, supervisors, prospective learners, and members of the instructional design team. Once formalized in writing, it should be reviewed periodically to make sure it remains current. Developing a Profile of Learner Characteristics. Instructional designers should be able to summarize the results of a learner assessment in a learner profile. A learner profile is a narrative description of the targeted audience for instruction that sets forth key assumptions that will be made about them as instruction is prepared. To be adequate, this learner profile should follow the results of the learner assessment and be complete enough to be used for making instructional decisions. What Should Be Included in a Learner Profile? A learner profile should clarify exactly what assumptions will be made about individuals who will, or should, participate in an instructional experience intended to rectify a performance problem. It can be thought of as a “role” (or even “job”) specification of the learner that summarizes the following characteristics. Necessary background knowledge, skill, attitudes, and physical traits. What should the learner already know or be able to do when he or she begins instruction? How should he or she feel about it? What minimum physical traits are necessary for success in the instructional experience?

Identifying the Target Population and Environmental Characteristics

75

Other necessary learner characteristics. These include any assumptions made about learners’ demographic or physiological characteristics, aptitudes, experience, learning styles, attitudes, job categories, value systems, life-cycle stages, or career stages. It is also wise to indicate reasonable accommodation that can be made for the physically or mentally disabled and those experiencing special learning problems. How Should a Profile of Learner Characteristics Be Developed? There are

three basic ways to develop a profile of learner characteristics for instruction: normatively, descriptively, and historically. The normative profile is established judgmentally, without considering the existing “market” of learners. Instead, it summarizes characteristics of the “ideal” or “desired” learner. To develop such a profile, instructional designers—or instructional designers working with operating supervisors and managers—may make arbitrary assumptions about what knowledge, skills, attitudes, physical traits, and other characteristics learners should possess before they enter instruction. The descriptive profile is established by examining the characteristics of an existing group and describing them. It summarizes characteristics of the probable or likely learner. To develop such a profile, instructional designers—working alone or in tandem with experienced job incumbents and supervisors—select a representative random sample of a “targeted group of learners” and describe their knowledge, skills, attitudes, physical traits, and other relevant characteristics. The historical profile is established by examining characteristics of those who participate in instruction. It summarizes characteristics of the historical learner. To develop such a profile, instructional designers should track the knowledge, skills, attitudes, and physical traits of those who participated in instruction and who then became exemplary (excellent) performers. With this information, it is possible to develop a predictive profile of those most likely to succeed following instruction. Acting Ethically in Assessing Relevant Characteristics of Learners A key ethical issue in assessing characteristics of learners can be expressed by this question: Is the learner assessment free of bias and stereotyping? In justifying assessments of learner characteristics, instructional designers should take care to avoid intentional or unintentional bias. One way to do that is to use sampling methods, keeping assessments strictly focused on learner characteristics that are essential to work success. Training is a

76

Mastering the Instructional Design Process

selection method. It must comply with the same legal safeguards as other selection methods (see Arvey and Faley 1988). Respondents to Rothwell’s (2003) survey indicated that assessments of learner characteristics can sometimes be a source of ethical challenges. One respondent wrote that, too often, assessments of learner characteristics are nothing more than “basic sociodemographic data of little relevance.” A second respondent pointed to problems that can surface when managers “feel that unit performance issues reflect poorly upon themselves personally.” A third noted, without explanation, that “learning styles are not valid in our work” and a fourth bewailed the tendency of his or her organization to “try to put into one method an approach to testing everyone.” These comments from the real world reflect key challenges faced by instructional designers as they try to apply theory and demonstrate instructional design competence. Applying Assessments of Relevant Characteristics of Learners Cross-Culturally Cultural beliefs can affect learners’ views about instruction. Targeted learners should be assessed for relevant cultural views that may affect instruction as it is designed, developed, delivered, and evaluated. Key cultural dimensions about people are listed in the left-hand column in the list depicted in Table 4.2. Important questions to consider about the targeted learners’ cultural views are listed in the right-hand column. By considering the list of cultural issues, instructional designers may more effectively tailor instruction to learners in a specific culture. Analyzing Relevant Work-Setting Characteristics Analyzing the characteristics of a work setting means gathering information about an organization’s resources, constraints, and culture so instruction will be designed in a way that is appropriate to the environment. For simplicity’s sake, we will call it setting analysis. Setting analysis is one form of analysis performed before instructional objectives and materials are selected or written. Tessmer (1990) suggested that environmental analysis is often neglected in the instructional design process. According to the author, “One reason for not including environment analysis is ignorance; the process is considered a simple procedure of asking a few questions about rooms and lighting for media projects” (56). We now know that environmental analysis is a critical part of the overall analysis process.

Identifying the Target Population and Environmental Characteristics

77

TABLE 4.2. IMPORTANT CULTURAL QUESTIONS Cultural Dimension

Important Questions to Consider

The individual

How much is individualism valued over groups or families? How widely do laws, rules, and regulations apply? Do they apply to everyone, or are exceptions made based on other considerations? How much respect is afforded to age in the culture? How is age regarded? Is increasing age associated with experience or with being out of touch? How is physical size and weight regarded in the culture? What are the physical requirements associated with the work, and how are those regarded in the culture? How well respected is education in the culture? How well respected is experience in the culture? How much does gender affect expectations about what people may or may not do, or who may or may not speak, and when?

Age

Height and weight

Education and experience Gender

According to The Standards, one competency for instructional design is to “identify and describe target population and environmental characteristics” (Koszalka, Russ-Eft, and Reiser 2013, 40). It is an essential competency. The performance statements associated with this competency indicate that instructional designers should be able to: “(a) Determine characteristics of the target population that may impact the design and delivery of instruction (essential); (b) Determine characteristics of the physical, social, political, and cultural environment that may influence learning, attitudes, and performance (advanced); (c) Identify the infrastructure that is available to support the design and delivery of instruction (advanced); (d) Determine the extent to which organizational mission, philosophy, and values may impact the design and delivery of instruction (advanced); (e) Analyze, evaluate, and use learner profile data and environmental characteristics to design instruction (essential)” (Koszalka, Russ-Eft, and Reiser 2013, 40–41). In this section, we explain the importance of setting analysis, we identify key environmental factors, and we describe how to carry out this form of analysis. We also offer advice about judging and justifying setting analysis.

78

Mastering the Instructional Design Process

Finally, we conclude by identifying key ethical and cross-cultural challenges and new developments affecting work-setting analysis. The Importance of Setting Analysis. In a classic description, Steele (1973)

emphasized the importance of physical settings in planned organizational change efforts. He noted, “If one attempts to make changes in the social functioning of an organization, one must pay attention to the physical systems which form part of the context for the social system” (6). As he defined it, physical system means setting or environment. It is “the total surrounding context for the person or the subject of interest” (6). Theorists have also more recently emphasized the importance of the setting in which people apply what they learn (Dubois and Rothwell 1996; Senge 1990; Watkins and Marsick 1993). Michael Allen (2012) emphasizes the important of context in designing learning interventions. The author notes: “In many ways, context is both the most foundational component (of instruction) and the most frequently neglected. When missing, it is much more difficult for learners to understand and remember situations that require one response instead of another” (25). In what Ruth Colvin Clark (2010) refers to as “Immersive Architecture” (also known as whole-task instructional design), learning is introduced in the context in which the new skills will be applied rather than in small, seemingly unrelated parts. By doing so, emphasizes Clark, “training relevance is more salient and transfer to the job may be more direct” (63). A dynamic interaction exists when knowledge is applied to a situation by an individual. Ignoring the environment and the situation in which knowledge is applied is metaphorically akin to focusing on a hammer but not considering the nail to be used or the board to which the nail is to be affixed. Those who set out to change organizations require what Steele (1973, 8) called environmental competence, meaning “(1) the ability to be aware of one’s environment and its impact; and (2) the ability to use or change that environment.” To demonstrate environmental competence, managers—or such other change agents as instructional designers— should “be more aware of the setting,” “ask themselves what they are trying to do there,” “assess the appropriateness of the setting for what is to be accomplished, ” and “make appropriate changes to provide a better fit between themselves and the setting” (8). Detailed examinations have been conducted to determine the competencies required to facilitate organizational change, and instructional designers increasingly find they must demonstrate these competencies and those linked specifically to instructional design work.

Identifying the Target Population and Environmental Characteristics

79

The instructional design process is a change effort intended to meet or avert deficiencies in knowledge, skills, or attitudes. It should therefore be carried out with due appreciation for the environments in which instruction will be designed, delivered, and subsequently applied. If this step is ignored, instructional designers may experience stiff resistance from managers and prospective participants as they prepare instruction. Worse yet, participants in instruction may later experience much frustration if, when they return to their job settings, they may not apply what they learned because their managers or coworkers do not support it. Identifying Factors and Carrying Out Setting Analysis. Instructional designers should make systematic examinations of the development, delivery, and application environments at the outset of the instructional design process. The development environment is the setting in which instruction will be prepared; the delivery environment is the setting in which instruction will be presented; and the application environment is the work settings in which learners will be expected to apply what they learn. Each of these environments should affect instructional development, delivery, and application. What Characteristics of the Development Environment Should Be Assessed, and How Should They Be Assessed? Begin a setting analysis by focusing ini-

tial attention on the development environment, since it will affect how the instructional design project proceeds. First, list characteristics of the setting that may affect the instructional design assignment. Examples may include any or all of the following characteristics in the left column and briefly described in the right column of Table 4.3. Other developmental characteristics of the setting may also be considered. For instance, Weisbord (1993, 754) suggests that organizations may be examined by asking questions about six key issues, and each question may be adapted to focus on developmental issues: (1) What business are we in, and how does instruction contribute to that? (2) How is the work divided up, and how does division of labor affect instruction? (3) Do all needed tasks have incentives, and what incentives exist for participating in—and applying—instruction? (4) Does the organization possess coordinating technologies, and does the instruction being designed also possess coordinating technologies? (5) How is conflict among people and technologies managed, and how is such conflict addressed in instruction? (6) How are these issues kept in balance in the organization, and what part does instruction play in maintaining that balance?

80

Mastering the Instructional Design Process

TABLE 4.3. ASSESSING DEVELOPMENTAL CHARACTERISTICS Developmental Characteristic

Brief Description

The (apparent) nature of the desired change The organization’s mission

The prevailing desire to improve consistency or change the way the organization functions. The primary reason for the organization’s existence. A short description of the organization’s products and service lines, customers, philosophy of operations, and other relevant characteristics that affect why the organization exists and how it interacts with the external environment. Fundamental beliefs about the way the organization should function with its customers, employees, the public, and other key stakeholders. Includes not only what management says “should be done” but also what is “really done.” Beliefs about what the organization should do in the future and assumptions about the environments in which it is or will be functioning. The way that duties and responsibilities have been divided in the organization—that is, reporting relationships. The difference between what is and what should be, stemming from lack of employee knowledge or skills or poor attitudes. The resources available for carrying out instructional development.

Organizational philosophy and perceived values

The organization’s goals and plans The organization’s structure Results of a needs assessment and analysis Resources available for the development effort—people, money, time, equipment, and facilities Preselected instructional design methods

Managers’ predisposition to approach an instructional need in a specific way, regardless of results yielded by analysis.

Second, determine how many characteristics may affect the present instructional design assignment and how they may, or should, affect it. Given the culture of the organization and the performance problem that instruction should solve, consider three major questions: 1. Based on what is known of the organization, how many of these characteristics relate to the present assignment? 2. How are the characteristics relevant? What is known about them?

Identifying the Target Population and Environmental Characteristics

81

3. How should information about these characteristics be used in such subsequent steps of the instructional design process as analyzing tasks? Writing statements of performance objectives? Developing performance measurements? Sequencing performance objectives? Specifying instructional strategies? Designing instructional materials? Evaluating instruction? Third, conduct a reality check to make sure that the most important developmental characteristics have been identified, their key implications noted, and the information recorded for use during the instructional design project. To do that, discuss the questions above with key decision-makers in the organization, other members of the instructional design team, and experienced or exemplary performers in the organization. Analyze their responses carefully and make the changes they suggest when warranted. What Characteristics of the Delivery Environment Should Be Assessed, and How Should They Be Assessed? Focus attention next on the delivery

environment, since it will affect how instruction is received by managers and employees of the organization. First, decide how the instruction will probably be delivered. While final decisions about delivery strategies are not usually made until later in the instructional design process, determine whether managers in the organization have predetermined notions—and justifications for them—about how instruction should be delivered, who should participate in it, when it should be delivered, why it should be delivered, and what needs or whose needs are to be met by it. There are many ways to deliver instruction. It may be delivered on or off the job; it may be delivered to individuals (through computer-based or web-based training, e-mail attachment, mobile devices or teleconference, print-based programmed instruction, self-study readings) or to groups. The choice of what to examine in the delivery environment depends on how instruction will be delivered. Most instructional designers and other training and development professionals, when asked about delivery, usually think first of the classroom, although that is not the best, least costly, or most effective alternative. When instruction is delivered on the job, relevant characteristics are the same as those in the list following for the application setting. When instruction is delivered off the job and in a meeting (informal) or classroom (formal) setting, relevant characteristics to consider may include any of the following in the left column and briefly described in the right column (Crowe, Hettinger, Weber, and Johnson 1986, 128).

82

Mastering the Instructional Design Process

Delivery Characteristics

Brief Descriptions

Learner involvement

The extent to which participants have attentive interest in group activities and participate in discussions. The extent to which participants do additional work on their own and enjoy the group setting.

Learner affiliation

The level of friendship participants feel for each other, and the extent to which they help each other with group work, get to know each other easily, and enjoy working together.

Instructor support

The help, concern, and friendship the instructor directs toward the participants. The extent to which the instructor talks openly with students, trusts them, and is interested in their ideas.

Task orientation

The extent to which it is important to complete the activities planned. The emphasis the instructor places on the subject.

Competition

The emphasis placed on participants’ competing with each other for successful completion of the tasks and for recognition by the instructor.

Order and organization

The emphasis on participants’ behaving in an orderly and polite manner and on the overall organization of assignments and classroom activities. The degree to which participants remain calm and quiet.

Rule clarity

The emphasis on establishing and following a clear set of rules and on participants’ knowing what the consequences will be if they do not follow them. The extent to which the instructor is consistent in dealing with participants who break the rules or disrupt the group in its activities.

Instructor control

The degree to which the instructor enforces the rules and the severity of the punishment for rule infractions. The number of rules and the occurrence of students’ getting into trouble.

Identifying the Target Population and Environmental Characteristics

83

Delivery Characteristics

Brief Descriptions

Innovation

The extent to which participants contribute to planning classroom activities, and the number of unusual and varying activities and assignments planned by the instructor. The degree to which the instructor attempts to use new techniques and encourages creative thinking by the participants.

Other characteristics may also be considered. Use these lists as a starting point for identifying important characteristics of the delivery environment and determining how they may relate to delivering instruction. Also refer to the lists in deciding how these characteristics should be considered while you analyze tasks, write statements of performance objectives, sequence performance objectives, specify instructional strategies, design instruction materials, and evaluate instruction. Conduct a reality check at the end of these steps and when instruction is subsequently delivered. What Characteristics of the Application Environment Should Be Assessed, and How Should They Be Assessed? Characteristics of the application envi-

ronment may affect the instructional design process just as much as, if not more than, characteristics of the development and delivery environments. The application environment should be considered before instruction is designed to maximize the likelihood that learners will transfer what they learn from instruction to their jobs (Baldwin and Ford 1988). Historically, instructional designers have seldom paid attention to the application environment, concerning themselves instead with results at the end of the instructional experience. One unfortunate result is that not more than 10 percent of the estimated $100 billion spent on workplace instruction in the United States each year produces on-the-job change (Broad and Newstrom 1992). If instructional designers have paid attention to the application environment, they usually know that learners are more likely to transfer what they learn from instruction to their jobs when conditions in the two environments are similar, if not identical (Thorndike and Woodworth 1901a, 1901b, 1901c). Basarab (2011) points out that the ultimate impact of training can be predicted if the learners believe in the value of what they’re being taught and their intention is to apply the skills or knowledge back on the job. These factors are

84

Mastering the Instructional Design Process

heightened by providing context-relevant training and building belief and intention assessment elements into the actual training experience. Any or all of the following characteristics in the left column and briefly described in the right column may influence on-the-job application of instruction (Crowe, Hettinger, Weber, and Johnson 1986, 146).

Application Characteristics

Brief Description

Involvement

The extent to which employees are concerned about and committed to their jobs.

Peer cohesion

The extent to which employees are friendly and supportive of one another.

Supervisor support

The extent to which management is supportive of employees and encourages employees to be supportive toward one another.

Autonomy

The extent to which employees are encouraged to be self-sufficient and to make their own decisions.

Task orientation

The degree of emphasis on good planning, efficiency, and getting the job done.

Work pressure

The degree to which the press of work and time urgency dominate the job milieu.

Clarity

The extent to which employees know what to expect in their daily routines and how explicitly rules and policies are communicated.

Control

The extent to which management uses rules and other pressures to keep employees under control.

Innovation

The degree of emphasis on variety, change, and new approaches.

Physical comfort

The extent to which the physical surroundings contribute to a pleasant work environment.

Additional characteristics of the application environment may also be worthy of consideration (Fitz-Enz 1984, 210).

Identifying the Target Population and Environmental Characteristics

85

Application Characteristics

Brief Description

Leader behavior

The supervisor’s way of dealing with people, work flow, and resource issues.

Work behavior

Work-related interactions with co-workers and supervisor.

Delegation

Extent to which and manner in which the learner’s supervisor delegates and encourages new ideas.

Worker capability

Skills, knowledge, experience, education, and potential that the worker brings to the job.

Strictness

Firm and equitable enforcement of the company rules and procedures.

Equipment design

Degree of difficulty experienced in operating equipment.

Job satisfaction

Each worker’s general attitude and amount of satisfaction with the job.

External influences

Effects of outside social, political, and economic activity.

Safety

The organization’s efforts to provide a safe and healthy working environment.

Self-responsibility

Workers’ concern for quality and their desire to be responsible.

Resources

Availability of tools, manuals, parts, and materials needed to do the job.

National situation

Impact of national conditions on the worker and the company.

Co-workers

Mutual respect and liking among members of the work group.

Pay and working

Performance reviews, promotions, pay, working conditions, and scheduling.

Job stress

Environmental effects such as temperature and ventilation, plus feelings about job security.

Personal problems

The impact of overtime on personal life and other issues concerning personal life. The sense of self-respect—and respect from others—that learners derive from doing the job.

Self-esteem

86

Mastering the Instructional Design Process

Application Characteristics

Brief Description

Work problems

Physical and psychological fatigue resulting from work.

The organization

General attitudes toward the organization, its style of operation, and its stability.

Economic needs

Degree to which the work satisfies workers’ needs for food, clothing, and shelter.

Responsibility accepted

Desired workload and responsibility versus actual workload and responsibility.

Organizational policies

Rest periods, training, work layout, and departmental characteristics.

Pepitone (2000) identified 10 characteristics of high-performance work systems that should influence the way we design and deliver training solutions: 1. Market-driven design and operation (influenced by customers and the external environment) 2. Optimized work systems (engineered for mechanical and human precision) 3. Clear direction and expectations (clear purpose, requirements, measures of performance) 4. Empowered and self-managing work teams (teams decide how to do their work and they manage themselves) 5. Capacity to redesign work processes (ability to redesign work to meet changing requirements and conditions) 6. Job flexibility and performance support (broaden skills and knowledge, facilitate learning, generate intrinsic motivation 7. Supportive management systems and culture (including decisionmaking, planning, budgeting, and information systems) 8. Supportive human resource policies (including such practices as skill-based pay, peer feedback, team bonuses) 9. Open access to information (access to all required information and the ability to disseminate information as needed) 10. Variance controlled at the source (the ability to detect errors at their source and correct them) (33–34) All of these characteristics carry with them a more effective way of preparing employees for the responsibilities of fitting into a highperformance workplace environment. More recent attention has focused

Identifying the Target Population and Environmental Characteristics

87

around so-called employee engagement, an organizational environment in which people feel a close connection emotionally between themselves and the organization’s mission, purpose, and goals. Concerns have been raised because a diminishing number of workers—only about 29 percent of workers in the United States—are fully engaged (Crabtree 2007). Research by Rothwell (2002) indicates that learners may face barriers in the learning process. Rothwell has published an assessment instrument that is useful in examining the learning climate to assess how well learners may learn based on organizational conditions, a key issue in facilitating individual learning and transfer of training. Use these lists of characteristics to analyze the application environment. First, determine how many characteristics relate to the present instructional design assignment and how they may (or should) affect it to improve the chances that instruction will subsequently be applied by learners on their jobs. Given the culture of the organization and the performance problem that instruction should solve, consider the following questions: • Based on what is known about the organization, how many of these characteristics relate to the present assignment? • How are the characteristics relevant? What is known about how each characteristic effects on-the-job performance? • How should information about these characteristics subsequently be used in the instructional design process to improve the chances that learners will apply on the job what they learned during instruction? How should this information influence subsequent steps in the instructional design model? As in the analysis of characteristics affecting the development and delivery environments, conduct a reality check to ensure these questions have been answered appropriately. In addition, make notes to use during the instructional design process. When necessary, recommend that managers make noninstructional changes to the work environment to encourage on-the-job application of learning. They also give the setting analysis legitimacy and grounds for justification. What Is New in Analyzing Relevant Learner and Work-Setting Characteristics A major issue that has emerged in the past several years has been accessibility. For instructional designers, this means that each step in the

88

Mastering the Instructional Design Process

instructional design process should consider the needs of a subset of the training audience neglected or underserved in the past. This population is broad and includes those with disabilities, those with cultural or language challenges, and those who might otherwise have been subject to discrimination. One major initiative related to providing effective training for everyone has been federal and state legislation guaranteeing equal access for everyone, regardless of physical or mental disabilities. The primary legislation affecting equality is the American with Disabilities Act, also known as the ADA. The law applies to all private employers, state and local governments, employment agencies, and labor unions. Since 1994, it covers all employers with 15 or more employees. Under the law, someone is considered “disabled” if their impairment limits one or more life activities. If an employee’s disability prevents him or her from performing a job, the employer must make a “reasonable accommodation” as defined by the law. Besides the ADA, protection against discrimination based on race, color, sex, national origin, and religion is guaranteed under the Civil Rights Acts of 1964 and 1991. What this means for employers is that no one can be denied training or training-related services based on disabilities or discriminatory factors covered under the Civil Rights Act. Beyond these requirements, designers increasingly find themselves in a position of designing instruction for those whose cultural and/or language experiences might serve as a roadblock to successfully completing the training. This has led to a re-examination of how we design and deliver training. One approach to designing instruction that takes these factors into account is often referred to as Universal Instructional Design (UID). The objective of UID is to approach instructional design from an inclusive, rather than an exclusive perspective. In the past, learners have been “accommodated”; that is, they were singled out for special treatment and the instruction itself, and the tools that supported the instruction (e.g., computers, audio files, e-learning programs) were “adapted” to their needs. UID principles encourage the designer to create an instructional experience that is accessible to all students regardless of language skills, ethnicity, abilities, or disabilities. Rush and Schmitz (2009) identify several core principles of UID: • Class climate: Adopt practices that put high value on diversity and inclusiveness. • Interaction: Encourage interaction between learners and instructor using methods that are available to all learners.

Identifying the Target Population and Environmental Characteristics

89

• Physical environments and products: Develop materials, activities, and equipment that are accessible to everyone. • Delivery methods: Use instructional methods that are available to everyone. • Feedback: Provide feedback regularly. • Assessment: Regularly assess learners on their progress and adjust instruction accordingly to meet everyone’s needs. • Accommodation: Plan to accommodate the learners whose needs are not being met by the instructional design. This approach has a profound effect on the entire ISD process. When conducting a needs assessment, we have to consider the broadest possible audience who may have a need, including those from other cultures (ethnic and organizational), those with language challenges, those with disabilities, etc. Once the needs are identified, our analysis must be broad to include those not formerly served by instruction. Most apparent, though, is the impact that UID has on the design, delivery, and assessment of instruction. As noted before, the UID principles and practices require us to develop instructional materials and methods suitable for the entire learner population rather than developing material for the majority of learners, then modifying or adapting it after the fact.

CHAPTER FIVE

USING ANALYTICAL TECHNIQUES TO DETERMINE INSTRUCTIONAL CONTENT

T

his chapter addresses the instructional designer’s skills and abilities in identifying content that matches the requirements as identified in the needs assessment phase. In today’s design environment, an increasing emphasis is being placed on analytical methods. Employers and clients expect instructional designers to do more than mindlessly plug content onto web pages or some other such low-level task for which little education or experience is required. Koszalka, Russ-Eft, and Reiser (2013) identify the following essential performance standards that support this phase of the instructional design process: “(a) Identify the scope of required content in accordance with needs assessment findings (essential); (b) Elicit, synthesize, and validate content from subject matter experts (essential); (c) Analyze existing instructional products to determine adequacy or inadequacy of content, instructions, and learning (essential); (d) Determine the breadth and depth of intended content coverage given instructional constraints (essential); (e) Determine subordinate and prerequisite skills and knowledge (essential); (f) Use appropriate techniques to analyze various types and sources of content (essential)” (Koszalka, Russ-Eft, and Reiser 2013, 42). As pointed out by the authors of The Standards, these skills involve higher-order cognitive skills, including the ability to critically analyze the information gathered to this point, synthesize, or bring together, relevant 90

Using Analytical Techniques to Determine Instructional Content

91

pieces of the puzzle in an efficient manner, and, finally, to evaluate the likely effect of the solution before the “go-live” decision. Fortunately, this phase of the process is more likely to succeed if the proper groundwork has been laid ahead of time. This groundwork would include: • A close working relationship with key stakeholders. • A thorough needs assessment that has delineated instructional and noninstructional needs. • A clear picture of on-the-job requirements and any constraints that might affect the application of newly acquired skills and knowledge. • The identification of knowledgeable people who understand the instructional design process and will work closely in identifying content. • A thorough understanding of existing resources (both training and nontraining) that have been, are being, or could be used to support the instructional process.

Identify Content Aligned with Needs Assessment Findings When selecting or developing instruction, always be sure to check whether the content aligns with the needs assessment findings. That advice sounds simple. But the reality is that it is easy to lose sight of goals when in the midst of reviewing large amounts of information in the form of preexisting courses from other sources or of feeling time pressures to get products out. One way to do that is to rely on a simple sequence. First, assess the needs. Second, clarify the instructional/performance goals or objectives. Third, prepare ways to measure the goals or objectives through tests or other methods. Fourth, prepare an outline of the instructional design that is intended to meet the needs by achieving the performance objectives/goals.

Elicit, Synthesize, and Validate Content from Subject Matter Experts Although a later chapter will cover formative evaluation (by which drafted instructional products are pilot-tested), an important early step in instructional design is to check signals with subject matter experts.

92

Mastering the Instructional Design Process

How do we know an SME when we see one? That is a common question. It has no simple answer. But an SME is someone who is qualified to judge a problem with human performance or a proposed solution (such as training) based on: • Education • Experience • Reputation inside and/or outside the organization gained by a track record of achievement. SMEs may be accessed formally or informally. A formal approach is to establish a task force or committee for the preparation of an instructional program. SMEs are appointed by the organization’s leaders based on their knowledge about the issue. An informal approach is to “ask around” for the “go-to” people who can help to solve a problem or address an issue. They may be asked to review an outline or look over draft materials to offer their advice. SMEs are not always easy to work with. The best in-house experts may be busy people who do not eagerly accept additional work. For that reason, the approval of their immediate supervisors or other organizational leaders may be needed to get the SMEs to cooperate in reviewing materials. In some cases it may be necessary to recognize or reward them for their work in helping to review or develop instructional materials.

Analyze Instructional Products to Determine the Adequacy of Content, Instructions, and Learning to Determine Content Coverage How does an instructional designer know that instructional products adequately address the content? Again, it is a simple question with no simple answers. There are two ways to do it. One is holistically; the other is systematically. In a holistic approach, an instructional designer simply asks one or several SMEs to look over draft instructional products and answer one question: do the instructional products appear to meet the organization’s needs? If the answer is “yes,” then no further action is required. If the answer is “no,” then the SME is asked to offer specific suggestions for improvement.

Using Analytical Techniques to Determine Instructional Content

93

If a systematic approach is used, the instructional designers will prepare a checklist for use by SMEs to review the match between needs, objectives, measurements (such as test items), and content. SMEs are typically asked to rate the quality by which needs are met; the objectives address the needs; the instructional materials will help learners meet the objectives; and the measurements will help evaluate learner achievement. It is best to develop a separate checklist for each component of instruction (such as lessons) rather than the overall “course” or instructional product.

Determine Subordinate and Prerequisite Skills and Knowledge Determining subordinate and prerequisite skills and knowledge really has to do with sequencing instruction. What should be treated first? There are at least nine approaches to sequencing performance objectives and thereby sequencing the skills and knowledge planned to meet those objectives: 1. 2. 3. 4. 5. 6. 7. 8. 9.

Chronological sequencing Topical sequencing Whole-to-part sequencing Part-to-whole sequencing Known-to-unknown sequencing Unknown-to-known sequencing Step-by-step sequencing Part-to-part-to-part sequencing General-to-specific sequencing

Let us describe each one. It should be noted, however, that recent writings on instructional design have emphasized the importance of elaboration in which ideas are introduced at a basic level and then gradually built upon, much like a “pebble in a pond” (Merrill 2002). Chronological Sequencing When performance objectives are sequenced chronologically, the content is arranged by time sequence with the presentation of later events

94

Mastering the Instructional Design Process

preceded by discussion of earlier ones. Chronological sequencing is typically used with history. Many academic experts who write college textbooks favor a chronological approach, beginning with the history of their discipline. Instruction is sequenced from past to present to future. Topical Sequencing When performance objectives are sequenced topically, learners are immediately immersed in the middle of a topical problem or issue. For example, today’s newspaper headline may be of topical significance to a given performance objective, and it could be used as a starting point for instruction. Learners are then led back in time to see how the problem originated. They may sometimes be led forward to see what will happen if the problem is not solved. This sequencing method is sometimes called in medias res, a Latin phrase meaning that instruction begins “in the middle of things.” Whole-to-Part Sequencing When performance objectives are sequenced from whole to part, learners are first presented with a complete model or a description of the full complexities of a physical object (such as an automobile engine or the world globe), abstraction (such as steps in a model of instructional design), or work duty (such as writing a letter). Instruction is then organized around parts of the whole. For instance, learners are then led through each part of an automobile engine, each nation on a world globe, each step in a model of the instructional design process, or each task that makes up the work duty. This approach to sequencing was first advocated by Ausubel (1962), building on the work of Gestalt learning theorists (see the description in Rothwell and Sredl 2000). Learners should be presented with an overarching logic to govern what they should know (Pucel 1989). In this way, they can see how each part relates to a larger conceptual system. Part-to-Whole Sequencing When performance objectives are sequenced from part to whole, learners are introduced to each part of a larger object, abstraction, or work duty. By the end of instruction, they should be able to conceptualize the entire object or abstraction or perform the entire duty. For example, learners

Using Analytical Techniques to Determine Instructional Content

95

could be oriented to an organization by visiting, investigating, and charting work activities in each department. They should eventually be able to describe the activities of each organizational part and thus (presumably) the entire organization. Known-to-Unknown Sequencing When performance objectives are sequenced from known to unknown, learners are introduced to what they already know and are gradually led into what they do not know. Herbart (1898) was among the first to advocate this approach to sequencing desired results of instruction, arguing that learners bring their experience to bear on what they learn. Consequently, he concluded that it is essential for instruction to build on what the learner already knows. Suppose, for example, that it is necessary to train a novice on how to make an overhead transparency on a copy machine. A trainer wishing to save time would first pose two questions: (1) Does the novice already know what an overhead transparency is? and (2) Does the novice already know how to make paper photocopies? If the answer to either question is no, instruction will have to begin by providing this essential prerequisite information. But if the answer to both questions is yes, the trainer can begin by explaining that transparencies are simply placed in the paper tray of a photocopier and an original sheet is copied. The result: an overhead transparency. In this way, the trainer has sequenced instruction from what the learner already knows about transparencies and photocopying to what the learner does not know about producing transparencies. Unknown-to-Known Sequencing When performance objectives are sequenced from unknown to known, learners are deliberately disoriented at the outset of instruction. In short, instructional designers consciously set out to “put the learners in over their heads.” It is sometimes called discovery learning. This approach dramatizes how little they really know about a subject or the performance of a task or work duty with which they already feel smugly familiar. The aim of this approach is to motivate learners for a subsequent learning task. It gives them an uncomfortable experience that leads them to question their own knowledge, thereby demonstrating to them that they need to learn more. Perhaps the most obvious example is military boot camp, in which new recruits undergo an initial upending experience that

96

Mastering the Instructional Design Process

clearly dramatizes how little they really know about their own physical and mental limitations. Step-by-Step Sequencing When performance objectives are sequenced step by step, learners are introduced to a task or work duty through either of two methods. The first method is based on the steps of the task or work duty itself. Instructional designers begin by analyzing how the task or duty is performed. They then sequence instruction around each step in the task or each task included in a work duty. The second method is based on the knowledge that learners must already possess or they must have mastered the skills to be capable of learning the procedure. Instructional designers analyze how people learn the skill or process information. This analysis is conducted using techniques such as information processing analysis, information mapping, or learning hierarchy analysis. Performance objectives are then sequencing around each step (“chunk of knowledge” or “specific skill”) that learners must possess to master a task or work duty. On occasion, training is not necessary for step-by-step learning to occur. Learners may be coached through a task by means of a job aid, such as a checklist or step-by-step description of a procedure. Alternatively, they may be coached through a task with a decision tool such as a flowchart, diagram, or electronic tool. Part-to-Part-to-Part Sequencing When performance objectives are sequenced part to part to part, learners are treated to a relatively shallow introduction to a topic, move on to another topic that is also treated superficially, move on to a third topic that is treated superficially, eventually return to the original topic for more in-depth exposure, and so on. The aim is to ensure that learners are introduced to topics and then hear more about them gradually as they are elaborated on in subsequent rounds of the spiral. General-to-Specific Sequencing When performance objectives are sequenced from general to specific, all learners are introduced to the same foundation of knowledge of the same skills. Later, however, each learner specializes. This method of sequencing

Using Analytical Techniques to Determine Instructional Content

97

is sometimes called the pyramidal or core structure method. All learners are exposed to certain topics (the core) but may specialize (by exposing themselves to topics around the core). Other Approaches to Sequencing Other sequencing methods may, at times, be appropriate. The nine described in this section are not intended to be exhaustive. They are, instead, intended to be representative of possible ways to sequence instruction.

Analyze Content Analyzing instructional content can be conducted in many ways. Perhaps best-known is so-called content analysis, which is not one method but many. Words may be analyzed for frequency. But a better approach is thematic analysis in which common concepts or themes are examined for frequency. A staple in qualitative methods, thematic analysis can be helpful in picking up common themes in words and is a counterpoint to statistical methods with numbers. Many software programs, such as NVIVO or N*UDIST, are widely used as tools to help analyze words. Analysis of this kind can be helpful when examining developed or subject-to-revision instructional products to determine “goodness of fit” to meet performance objectives. It can also be helpful in examining text stemming from interview data gathered during needs assessment, focus group data, secondary data presented in words, or similar information.

CHAPTER SIX

ANALYZING THE POTENTIAL USE OF EXISTING AND EMERGING TECHNOLOGIES

I

n the past half-century or more, the instructional design field has been increasingly influenced and altered by the advance of technology. In the 1960s and 1970s, instructional television had a huge impact on the design and delivery of instruction. In the 1980s and beyond, the personal computer provided infinite flexibility in delivering instruction. While there has always been controversy over the benefits-versus-costs of these technologies, the field will not go backward, only forward. With that in mind, no one who claims to be an instructional designer can plead ignorance of technology in the design, delivery, and evaluation of instruction today. Siemens (2005) posed seven compelling questions that must be addressed as advanced technology transforms the learning process: 1. How are learning theories affected when knowledge is no longer acquired in the linear manner? 2. What adjustments must be made with learning theories when technology performs many of the cognitive operations previously performed by learning (information storage and retrieval)? 3. How can we continue to stay current in a rapidly evolving information ecology? 4. How do learning theories address moments where performance is needed, absent complete understanding? 98

Analyzing the Potential Use of Existing and Emerging Technologies

99

5. What is the impact of networks and complexity theories on learning? 6. What is the impact of chaos as a complex pattern recognition process on learning? 7. With increased recognition of interconnections in differing fields of knowledge, how are systems and ecology theories perceived in light of learning tasks? (p. 3) These and other questions are addressed as new advancements are made. Technology is a topic of perennial interest, though it dates rapidly (as an illustration, see Rothwell, Butler, Maldonado, Hunt, Peters, Li, and Stern 2006). This chapter provides a foundation around which decisions can be made regarding the use (and abuse) of modern instructional technology. The evolution of technology-based training is depicted in Figure 6.1. What can be said about the technologies of the 1970s and 1980s is that they were expensive to develop and posed a huge challenge in keeping the content updated and relevant. In addition, the technology was difficult to deploy, did not adhere to common technology standards, and did not lend itself well to tracking individual and group progress. This latter challenge led to the development of learning management systems (LMS) and SCORM, the specifications for tracking and structuring content. More about each of these concepts later in this chapter. As in many other disciplines, an intelligent and balanced approach to selecting instructional delivery tools is warranted. No one solution is always “best”; the business goals and the context for learning and application must be factored into any instructional delivery decision. However, as Caruth and Caruth point out (2012), the workforce of the twenty-first century “needs to be comfortable with critical thinking and problem solving, communication, collaboration, creativity and innovation” (p. 713).

FIGURE 6.1. TECHNOLOGY-BASED LEARNING TIMELINE

Mainframe-Based Training Instructional video 1960s– 1970s

Computer-Based Training Web-Based Training Distance Learning E-Learning CD-ROM Virtual Classrooms 1980s– 1990s

1990s

2000

Blended Learning MOOCs Wikis Social Media Mobile Learning Present

100

Mastering the Instructional Design Process

Today’s technology can go a long way in delivering these requirements if used properly. For the balance of this chapter, we will focus on the technologies prevalent in today’s instructional design world, including e-learning, computer-based training (CBT), web-based training (WBT), wikis, mobile learning, and social media. The evolution that has taken place over the past two decades or more centers about the delivery of training solutions through the World Wide Web called “the web.” This ubiquitous technology has passed through two stages and is entering a third stage. To understand this evolution and how it has and is affecting the field of instructional design, we need to briefly visit each stage of the process.

E-Learning In today’s world of training, the term e-learning has become synonymous with a variety of computer-based instructional delivery systems. E-learning refers to instruction delivered via the Internet, a corporate intranet, a network, or via CD-ROM (although this last medium is fast disappearing). Recent advances in computer technology have allowed e-learning to achieve high levels of interactivity while appealing to all five senses. E-learning can be synchronous or asynchronous. Synchronous learning is done in real time with a live instructor interacting with a group or groups at different locations. This learning environment allows for additional interactivity including instant messaging, application sharing, and participant polling. In addition, social media platforms such as Twitter allow participants to ask questions or make comments in real time. This allows the leader/instructor to make adjustments to the delivery on the fly. Asynchronous learning is self-paced and allows learners to advance through a course quickly or as slowly as they choose. There is no real-time interactivity between learner and instructor as with synchronous environments. Common asynchronous tools include blogs, learning forums, and webcasts. These two approaches (synchronous and asynchronous) can be combined within a single training or education program, as with many university online courses today, where content is delivered asynchronously and group discussions or question sessions can be done synchronously.

Analyzing the Potential Use of Existing and Emerging Technologies

101

E-learning is the foundation of both computer-based training (CBT) and web-based training (WBT). Computer-based training is primarily asynchronous and it packaged in CD or DVD formats. Web-based training is delivered over the Internet and can be both synchronous and asynchronous in design. Arshavskiy (2013) points out that there are four levels of interactivity when dealing with e-learning platforms. These include: Level 1: Passive interactions: The learner has no control over the progress of the learning: they watch and read as the program progresses. While relatively inexpensive to develop, it lacks the motivation of more advanced methods. Level 2: Limited interactions: Here, the learner has more control over the sequence of the program. It might also include questions, hyperlinks, activities, audio, and video. This level is appropriate for teaching lower level content such as facts and procedures. Level 3: Complex interactions: Besides learner control, the program may include animation, simulations, or scenarios. This level is appropriate for skill development or problem-solving challenges. Level 4: Real-time interaction: This level might include all of the above characteristics plus complex content, serious learning games, avatars, and 3D simulations. This level is good for complex problem solving or application of skills and knowledge to real-world situations. The learning and motivational potential of today’s technology is staggering. One only must spend a few minutes with a PlayStation or X-Box game to see the visual clarity and interactive potential of computer-based media. The challenge with e-learning, as with many earlier technology platforms, is knowing when and which ones to use in a situation. Arshavskiy (2013) summarizes the authoring tools available for today’s instructional designer. See Table 6.1. These tools are likely to be enhanced and, sometimes, replaced as the technology improves and the field becomes more comfortable with their use. One thing prospective instructional designers should know of is that most instructional designer jobs, whether internal or consulting, require at least a working knowledge of one or more of these tools. This is true even for management-level positions, so be advised that you should develop a level of skill with the most popular of these tools.

102

Mastering the Instructional Design Process

TABLE 6.1. AUTHORING TOOLS Category

Product

Description

E-Learning Course Development

Adobe E-Learning Suite

PowerPoint Plugin Authoring

Articulate Studio

Desktop Authoring

Articulate Storyline Adobe Captivate

Captivate—create simulations Flash—graphics, animation, games Dreamweaver—website development Photoshop—editing graphics Acrobat—viewing, creating, printing, and managing PDF files Presenter—creating video and converting courses into interactive e-learning presentations Audition—audio recording, editing, and mixing Presenter—record and synchronize narration, add web objects, quizzes, interactivity, and flash movies Engage—develop interactivities using preprogrammed templates Replay—recording screencasts with webcam and adding media. Create unique, fully customizable courses

Lectora

Create multimedia presentations in the form of movies, step-by-step tutorials, and animated help file enhancements Produce high-quality interactions and create branched learning scenarios

Computer-Based Training (CBT) Computer-based training is a form of e-learning in which training programs are delivered via a computer. Such courses are typically packaged in a CD (compact disc) or DVD format and are delivered asynchronously. The big advantage of CBT is its ability to be mass-produced and distributed to many users. The content is static and may require considerable time and money to update. Web-Based Training (WBT) Web-based training is delivered over the Internet and can be provided in both asynchronous and synchronous formats. Unlike CBT, which relies on large audio and video files whose performance degrades when running

Analyzing the Potential Use of Existing and Emerging Technologies

103

on the Internet, web-based training relies on HTML and browser-based platforms such as Adobe Flash. WBT also takes advantage of search and linking capabilities that are unique to internet technology. One major issue is the cost of developing e-learning compared to the development costs of other forms of instruction. Moran (2011) indicates that, in his experience, one hour of e-learning instruction can require, on average, one hundred hours of development, depending on the computer-based interactions and the multimedia requirements. Brian Chapman (2010) compared the development time and costs for e-learning and instructor-led training (ILT). This extensive study collected data from 249 organizations representing 3,947 learning development professionals who had created both ILT and e-learning content distributed to over 19,800,000 learners. A summary of these findings appears below. • The average time required to develop instructor-led training ranges from a low of 22 hours-to-one hour of instruction for rapid development, simple content, and repurposing of existing source material with minimal print-based learning support, to 82 hours-to-one hour of instruction for complex projects, with extended time spent on formatting during production. • Level 1 e-learning ranged from a ratio of 49:1 (rapid development, simple content, specialized authoring tools, including PowerPoint) to 125:1 (complex projects, difficult to produce, more media production). • Level 2 e-learning ranged from 127:1 (rapid development through template interactions, simple animation, and efficient or low-end media production) to 267:1 (advanced and custom interaction, embedded simulation activities, and extensive media). • Level 3 e-learning ranged from 217:1 (template interactions, games and simulation, efficient simulation development practices) to 716:1 (complex projects, advanced learning simulations and games, extensive media production). Blended Learning As the technologies we’ve just discussed matured, instructional developers asked questions about which technology was best for a situation and whether the technology provided a superior learning environment to instructor-led training. As with most advanced technologies, the answer was “It depends.” An approach known as blended learning became the focus in many instructional design circles.

104

Mastering the Instructional Design Process

Blended learning, as its name implies, is a combination of technologybased and non-technology-based (instructor-led training) approaches to learning and performance. The concept encourages instructional designers to avoid the trap of believing that a single solution (e-learning) is always the best approach. The goal of any instructional intervention should be to increase the productivity of those experiencing the intervention. Josh Bersin (2004) cites a dramatic finding when he points out that “A 1 percent increase in productivity has more than 10 times the financial impact as a 1 percent decrease in training costs” (p. 21). The implications of this finding are huge. They point to the fact that any training solution, technology-based or otherwise, should be judged primarily on its impact on performer productivity. This serves as a counter to the argument that technology-based solutions that can be distributed to a large number of participants efficiently are, by that fact alone, more desirable than other solutions because they reduce training costs. Bersin argues that, rather than looking exclusively on what training technology is most effective in cost reduction, we should use productivity gains as the ultimate measure of our success and the chief criterion for selecting a delivery platform for our instruction. By doing so, we can make an informed and research-based decision about what “blend” of instructional solutions best fit a situation. Table 6.2 shows examples of situations that suggest the proper blend of solution tools. From the same research cited earlier in this chapter (Chapman 2010), it was reported that the average size of the major components of a blended learning program comprising ILT, e-learning, and virtual classroom (co-located participants) was distributed as shown in the following list. • 44 percent of the typical blended learning program was made up of ILT. • 38 percent was made up of e-learning. • 18 percent was made up of virtual classroom time. Research contributors indicated that the average duration of their blended learning courses was 32 hours. Besides the data reported above, Chapman asked the survey contributors to estimate the average cost per finished hour to develop the various learning solutions. Here is what he found: • • • •

Instructor-led training (ILT): $5,934. Level 1 e-learning (Basic): $10,054. Level 2 e-learning (Interactive) $18,583. Level 3 (Advanced): $50,371.

Analyzing the Potential Use of Existing and Emerging Technologies

105

TABLE 6.2. BLENDED LEARNING TOOLS Learning Situation

Desired Learning Interactivity

Blended Solutions

The need to broadcast information (e.g., a new product announcement or corporate change) The need to transfer critical skills to the job

Read, listen, or watch

Conference call, webinar, e-mail, mobile device

Read, listen, watch, and ask or answer questions

Develop skills and competencies

Read, listen, interact, practice, ask/answer questions, take an exam, and get feedback Read, listen, interact, practice, answer questions, take exam, get feedback, pass or fail

E-learning, webinar, courseware, tracking completion of the content Web-based courseware, instructor-led class, simulations Web-based courseware, instructor-led class, simulation, LMS tracking to completion

Certification

Source: Adapted from J. Bersin, The Blended Learning Book. (San Francisco: John Wiley & Sons, 2004).

One last comment about cost in relation to these various approaches: Vance (2010) suggests that training functions should apply return-oninvestment (ROI) concepts (and other financial indicators) when deciding where and how to spend its resources. He argues that programs can be best justified when aligned with the highest priorities of the company. When considering which delivery options to use (instructor-led, web-based training, or a combination of both), consider the ROI of each approach as part of your decision-making process. Computing ROI can be an involved process, but, according to Vance, will help the training department to function as a real business unit. Having looked at the categories of electronic-based learning, or e-learning, we turn our attention to the platforms on which these technologies are delivered. We categorize the technologies into one of three domains (Web 1.0, Web 2.0, and—now emerging—Web 3.0). To begin with, we’ll define each of the web platforms, then look at how the technologies discussed earlier, specifically e-learning, fit into these platforms. Web 1.0: Static Content The term Web 1.0 first appeared in the literature in the early 1990s and refers to the initial period of providing learning content through the World Wide Web. It can be a “read-only web” in which interaction between the

106

Mastering the Instructional Design Process

learning application and the learner was dramatically limited or nonexistent and the learner had little or no capability to contribute to the content. Brian Getting (2007) described this period as “brick-and-motor thinking applied to the web” (p. 1), by which content providers used the web to provide their information to anyone while still supplying static content, much as you would find in a traditional bookstore. Web 1.0 e-learning provided static content to the learner through web portals (Google, Netscape, or Internet Explorer), but with limited capability by the learner to interact with or add to the content. As has been noted above, many of the limitations of static e-learning programs have been countered by the blended learning concept, in which the blend of one-way (asynchronous) learning, interactive (synchronous) learning, and instructor-led training can be brought together to produce the desired learning outcome. The rapid pace of technology, however, has provided an enormous opportunity to advance learning and performance. This evolution has moved the field beyond Web 1.0 to the next (and current) iteration: Web 2.0. Web 2.0: Mobile and Social Learning With the advance of Internet technologies and smaller and smaller devices, the training field has found dramatic new ways of delivering information where it is needed, when it is needed, and to those who need it in a useful form. Real-time content customization and input from a variety of stakeholders has become the norm. The advances described in the next section can be grouped under the term Web 2.0; that is, an extension of e-learning and other similar technologies far beyond anything dreamed of a decade ago. Web 2.0 encompasses applications that can be accessed from anywhere (your home computer, your cell phone, or tablet). In addition, Web 2.0 tools allow and encourage people to create and share information with each other across multiple platforms. For instructional designers, this new frontier offers exciting and challenging possibilities. Moore (2008) describes this evolution this way: “Web 2.0 tools have a knack for giving voice to the range of opinions, bringing people together and fostering a collaborative environment” (p. 2). Let’s examine some of the Web 2.0 applications growing in popularity. Wikis refers to any collaboration website in which members of a community can contribute to and edit content. The first and best known was Wikipedia, an online encyclopedia in which readers can add

Analyzing the Potential Use of Existing and Emerging Technologies

107

content and have it updated (or refuted) by other writers in real time. Some organizations use wikis to share ideas or notes and encourage responses and edits as the content or discussion thread spreads. Blogs are online journals or discussion threads that pertain to a topic or body of knowledge. A blog is usually managed by an individual and can be used in instructional settings to post questions, comments, assignments, or other relevant information. Mobile learning. With smartphones, tablets, and e-readers, the ability to send or allow access to information to anyone anywhere becomes real. Virtually every conceivable type of information can be accessed this way, including technical data, product information, sales data, forms, podcasts, videos, and much more. The information can be delivered in real time to virtually an unlimited number of people through a technology we use every day. While there is limited research data on the effectiveness of mobile learning across multiple uses, anyone with a smartphone today realizes the power of the device in accessing and disseminating information. Podcasts and vodcasts. Podcasts are audio recordings that can be accessed through mobile devices such as smartphones and tablets. Vodcasts are video podcasts available on the same platforms. Both are powerful tools for disseminating information or reinforcing content that may have originated in another medium. Professional organizations such as the Association for Talent Development (ATD) use podcasts extensively to introduce new ideas or provide a platform for deeper explorations into important topics. Social Media While the term “social media” can include a wide variety of technologies, some of which (blogs, wikis) we’ve discussed above, we’ll turn our attention to the two best known social media tools (Twitter and Facebook) and examine how they can be used by instructional designers to enhance and reinforce learning. Twitter originally limited a user to publishing “tweets” of only 140 characters, but that limitation is expected to expand in the near future. As a user, you can choose the people who will receive your tweets: other people whom you want to follow on Twitter and those who can follow you. Its popularity is so widespread that the word has become a verb (“I tweeted you today—did you get it?”). As a training tool, Twitter has unlimited possibilities. A trainer can manage discussions and assignments in real time. Some trainers encourage the

108

Mastering the Instructional Design Process

use of Twitter during a training session so participants can “tweet” questions in real time for immediate response. You can also attach articles and web links to tweets. That you’re limited to 140 characters in a single tweet encourages the sender to be concise and to the point. Perhaps its real potential as a training tool lies in its ability to build and sustain a community of users. A course on presentation skills might conclude by establishing a community on Twitter. After participants return to their location of origin, everyone (instructor and participants) can continue to share new experiences, videos of strong (or weak) presentations, articles or books they read, and so forth. The possibilities are endless. Facebook. The number of Facebook users worldwide is staggering! Like Twitter, it’s not at all uncommon today to end a discussion with something like “Be sure to put that on Facebook.” While there are concerns expressed about privacy and how Facebook and Twitter have altered our interpersonal communication styles, the potential of the technology to enhance learning and performance cannot be ignored. As with Twitter, Facebook allows users to communicate assignments, share useful information, post notes and documents, hold real-time chats, and much, much more. As Jane Bozarth points out (2010), Facebook can even substitute for many of the common learning management systems such as Blackboard and Moodle and even many of the corporate LMSs. Facebook offers a more expanded conversation thread than Twitter, which is limited to 140 characters, and Facebook is more interactive than blogging. The most attractive feature for instructional designers is that Facebook is ubiquitous. The majority of training participants will likely have a Facebook account and be comfortable in using the tool. This reduces the learning curve while providing a dynamic platform for after-training experiences. The challenge we face today and into the future is how to integrate all solutions to produce the desired results. And applied properly, they open up vast possibilities not only in training the so-called “hard” skills but the “soft” skills. Steve Fiehl (2012) points out that “the possible and desirable links between training, formal learning, business social networks and informal learning will create even more value for each person and for the business” (p. 27). He emphasizes the power of social networking (blogs and forums) to create a community of learners and sharers and provide the capability for a learner to learn on his or her own. It’s up to the instructional designer to blend these solutions into a package that delivers value efficiently and effectively.

Analyzing the Potential Use of Existing and Emerging Technologies

109

Social media has the potential to transform business practices well beyond the training arena; the technology has given rise to the term “social business.” Kiron, Palmer, Phillips, and Berkman (2013) describe the concept as an organization’s use of any of the following elements: 1. Consumer-based social media and networks (Twitter, Facebook, Google, YouTube). 2. Technology-based, internally developed social networks (Cisco Learning Network). 3. Social software for enterprise use, whether third-party developed or in-house developed. 4. Data derived from social media and technologies (crowdsourcing). In terms of social media’s potential to influence learning and development, organizations are seeing the light. Emelo (2012), citing a study done by Bersin by Deloitte, reports that U.S. companies spent on average $13,675 on social learning tools and services in 2012, an increase of 39 percent since 2011. This indicates the growing realization that social media and learning has the potential to transform learning and performance improvement in the years to come. Gamification In what has become almost a cross-over between the video gaming world and technology-based learning, the concept of gamification is becoming more and more popular. Training Industry (2014) defines gamification as “ … the process of applying gaming designs and concepts to learning or training scenarios in order to make them more engaging and entertaining for the learner.” The technology used to provide these game experiences can range from a simple and relatively inexpensive 2D flash platform to an elaborate 3D immersive experience. It has become especially attractive when used with the computer-savvy millennial generation. David Maddocks, the president of WorkSmart Education, is quoted in Business Education as saying “the basic structure of video games—having to master one level before moving to another, repeating an action numerous times, and receiving feedback in the form of results about what works and what doesn’t—mirrors how skills are developed in real life” (Osak 2014, 181). Meister (2013) points out that firms such as the systems integration company NTT DATA and the consulting firm Deloitte use gamification in their leadership development programs to experience the feeling of

110

Mastering the Instructional Design Process

being a leader. NTT DATA uses the Ignite Leadership game as a way of having leaders collaborate online, get feedback from their peers, and be recognized for their game play. The game addresses key leadership skills including negotiation, communication, time management, change management, and problem solving. While it’s difficult to predict the future, Zinger (2014) notes: “Perhaps before 2020 gamification proficiency will be a required competency for managers in most organizations” (p. 3).

The Effectiveness of Today’s Instructional Technology As with every advancement that has presented itself to instructional designers, the potential can sometimes exceed its performance. The challenge is to avoid the law of the hammer (when all you have is a hammer, everything looks like a nail!) and take an objective look at what sound research supports the promises for new technology. In this section, we will examine what the research indicates in selection, application, and effectiveness. While it is tempting to apply these new technologies to a variety of solutions, we need to base future decisions on the evidence, scarce though it may be. Here are some of the findings that might help to guide us in this quest. Ellis and Kuznia (2014) reported on a study they conducted that examined the impact of corporate e-learning on employees. The dependent variables in the study were employees’ productivity, job satisfaction, overall job performance, and organizational commitment. The authors collected their data from 10 educational institutions and two international corporations. Although limited by the data being self-reported, the following results were noted: 1. 91 percent of respondents reported that they had used e-learning. 2. 75 percent reported that e-learning leads to higher employee satisfaction. 3. 42 percent reported that their job responsibilities have been easier because of their e-learning experience. 4. 73 percent said that e-learning leads to higher employee productivity. 5. 66 percent believed that e-learning leads to higher employee organizational commitment. While this data is encouraging, it is tempered by it being selfreported. Little research exists that demonstrates the effectiveness of technology-based learning on actual job performance—another

Analyzing the Potential Use of Existing and Emerging Technologies

111

challenge in evidence-based research practices. While we might measure effectiveness to acquire new learning (we can do that through a simple test), skill and knowledge transfer is more difficult to demonstrate, given all of the variables that affect the outcome. We can establish measures of efficiency when comparing e-learning solutions with classroom (ILT) approaches. Kasraie and Kasraie (2010) suggested two formulas for computing cost efficiency: • TCTP – TCNP = PNS, where TCTP represents the administrative costs of the former program, TCNP represents the total administrative costs of the new program, and PNS equals the projected cost savings. • TACT/number of students = CPS, where TACT is the total administrative costs of training and CPS equals the cost per student. Using these formulas, one can see that e-learning is more efficient than classroom training given travel costs and such variable costs as building maintenance and utilities. The authors admit, though, that such calculations do not determine the effectiveness, which can only be determined through standard evaluation procedures. Mayer (2008) suggests that we need to consider whether a proposed instructional method follows what we know from modern research about how people learn. One problem he cites is that what we know about how people learn doesn’t easily lead to knowing what instructional methods and materials to use. The author urges the field to focus more on authentic learning situations (that is, what happens in practice) as opposed to contrived situations conducted in the laboratory. Only by doing this will we be more confident about what works in the real world. What’s Coming Next? While we may not know what the instructional design landscape will look like five years from now, or even next year, the technology that’s in use today will find its way into the learning and performance mainstream. When we look back at the technologies that dramatically affected human progress in the past 50 to 75 years (especially television and the computer), one common thread ran through all of them. After proof-of-concept and early scientific adoption, insightful people asked the same question: “Can we use this technology to educate people?” As in the past, many of the current technologies are expensive and not widely available, but that changes as it has with past advances.

112

Mastering the Instructional Design Process

This section focuses on describing Web 3.0. Of particular interest to the instructional designer, though, is not only the technology itself and its potential to drive learning and performance improvement but the implications for learning design itself. Before discussing this topic, though, let’s define this world we’ve just alluded to and provide examples. Web 3.0. The rapid advance of technology begs the question “What’s next?” While we won’t know until we get there, the possibilities of these advances for improving learning and performance are endless. Michael Green (2011) provided an insight into what the future may look like in his definition of Web 3.0. While Web 1.0 provided static, predetermined content and Web 2.0 introduced interactive technology that we’re comfortable with (Twitter and Facebook), Web 3.0 is a giant leap forward. Green describes Web 3.0 as having three main components: 1. The semantic web, in which software can understand the meaning of data and translate that understanding into natural language, creating a customized experience that meets the user’s needs where he or she is. We’ve all had the experience of doing a Google search for a certain topic, only to have over a million hits returned! Most of us don’t have the patience to scroll through more than one or two pages before we make our selection, usually out of frustration. Even if we refine our search, we may find exactly what we were looking for. The semantic web understands natural language and “knows” what we’re looking for, getting us to our destination much faster and more efficiently. 2. The mobile web, whereby the user can experience the web seamlessly, from one device to another, in any location anywhere in the world. This advancement provides us with a consistent interface and application platform regardless of whether we’re using our smartphone, tablet, desktop, or any device that can access the web. 3. The immersive Internet, comprising virtual worlds, augmented reality, and 3-D environments. This capability is best known today in the video gaming world, but its applications for training are incredible. Up to now, skills-based training was limited in its ability to recreate the actual environment in which the new learning will be applied. With the immersive Internet, we can create a “virtual” world that contains nearly all of the physical, mental, and tactile conditions of the work environment.

Analyzing the Potential Use of Existing and Emerging Technologies

113

These advances should provide boundless opportunities for the instructional designer of the future in providing learning experiences as a part of one’s daily life, using devices as comfortable to us as our favorite easy chair! This author wonders what will be written about in the next edition of this publication. Whatever it is, it will surely stir the imagination and open new vistas for learning and performance improvement specialists. The Implications of Learning Technologies for Instructional Designers Just as the field has grown and changed as new theories of learning emerged (behaviorism to cognitivism to constructivism), so too has technology influenced the development of today’s instructional designer. In the early years of e-learning and web-based learning, the instructional designer could rely on the “technologists” to produce the final product. In today’s world, however, a lack of hands-on experience with today’s most widely used tools (Captivate, Adobe Flash, and so forth) has become a liability. Even a Project Manager or Lead Instructional Designer position usually requires experience with these and other applications. But it’s not only the technology itself or the applications that require additional knowledge by the instructional designer. E-learning development also employs several other skills not necessarily well known to instructional designers who have lived exclusively in the print-based or instructor-led world. Let’s look at where the print-based/ILT world and the e-learning world converge and where they deviate. Though not an in-depth study of the topic, this section should provide guidance for the reader in knowing what he or she might need to become a well-rounded designer. Commonalities. Whether designing instruction in the print-based or the e-learning worlds, the designer must have a solid foundation in the following four skills. 1. Instructional design models, from ADDIE to SAM (Successive Approximation Model) and beyond. The selection of the model based on a variety of factors will affect the success of the project. 2. Learning theories. While the technology may be advanced, the designer must be grounded in a variety of learning theories, from behaviorism to constructivism, and which theory best fits the needs. 3. Evaluation and assessment. The designer must be able to create and “sell” a process for evaluating the program under development across

114

Mastering the Instructional Design Process

multiple levels (Level 1: reaction to Level 5: ROI) depending on the project. Besides evaluation, the designer must be comfortable with assessment concepts; for example, reliability and validity, test item construction, principles of feedback, etc. 4. Project management and consulting skills. All designers must know how to create a design document that identifies the key elements of the project. They then need to manage the project, considering any change and risk management challenges that might arise. In addition, strong consulting skills, including questioning, listening, trust-building, and ethical practices, need to be present. Differences. While some skills may overlap the print-based/ILT development world, the following five are essential for the e-learning developer. 1. Storyboarding. Storyboarding is not a new phenomenon: Gone With the Wind (1939) was the first full-length motion picture to be developed using the technique. While print-based developers have used storyboarding often, the approach is a fundamental tool for the e-learning developer, primarily because he or she is attempting to integrate several instructional tools, including audio, content, graphics, and instructions to the programmer about how the end product should flow. 2. Rapid prototyping. Besides storyboarding, this is another core technique in e-learning development. As Arshavskiy (2013) explains, there are three prototyping styles: • Nonfunctional prototypes: While they lack functionality, they provide the “look and feel” of the intended functionality that will be in the final product. • Semi-functionality: Selected interactions that can be used in screenshots in storyboards. • Fully functional: Include most of the content, functionality, and assessment and represent the full functionality of the final product (p. 86). 3. Script writing. Unlike ILT instruction, e-learning relies heavily on scripts that must support the final product. As in script writing there are unique skills required to make the script comply with the needs of e-learning. It should be easy to listen to, and the content should be concise, to-the-point, and engaging to listen to. 4. SCORM compliant. This standard (Shareable Content Object Reference Model) ensures that an e-learning course will function with other

Analyzing the Potential Use of Existing and Emerging Technologies

115

e-learning software and compliant learning management systems, or LMS. 5. One important consideration in developing e-learning is the requirement that it be accessible by those with learning disabilities. Section 508 of the Rehabilitation Act requires that all e-learning courses must meet the needs of those with auditory, visual, and motor disabilities. These requirements are fairly extensive, but here is a sample of what developers must consider: • All audio and video files must offer synchronized captions for the disabled. • All information presented in color must also be available in black and white. • Avoid elements that flash. • Provide consistent navigation features. • Provide text-only pages with information that is equivalent to the nontext version. The rapid growth of technology-based instruction demonstrates that it is here to stay and will take advantage of technological advances coming at a rapid pace. As instructional designers, however, we need to assess the effectiveness of this growing menu of options to provide the most cost-effective learning solutions for our clients. It should go without saying that today’s instructional designer has a greatly expanded number of responsibilities to fulfill. Technology has always been a part of our field but, in the past, designers could either avoid learning the ins-and-outs of what existed at the time or find someone else to do the work. Today, however, the designer cannot ignore the dramatic advancements in technology; no designer can consider himself or herself to be a professional without at least a working knowledge of what’s available and how it should be used effectively. Nowhere is this more evident than in the job listings for instructional designers. Even management-level people must have experience with e-learning tools such as Captivate or Flash. In addition, with social learning tools, the ability to provide learning opportunities has dramatically expanded. What we’ve learned in the past continues to provide a solid foundation for our field. Such elements as the learning hierarchies, the nine elements of instruction, and how we organize knowledge will continue to inform our work. With technology, we have the potential to take advantage of what we know about learning and provide more customized, efficient, and engaging learning environments for those we serve.

PART THREE

DESIGNING AND DEVELOPING FOR INSTRUCTIONAL DESIGN

Part VI: Predicting the Future of Instructional Design

Part I: Overview

Part V: Managing Instructional Design

Part II: Planning and Analyzing for Instructional Design

Part IV: Evaluating and Implementing Instructional Design

Part III: Designing and Developing for Instructional Design

CHAPTER SEVEN

USING AN INSTRUCTIONAL DESIGN PROCESS APPROPRIATE FOR A PROJECT

A

ccording to Instructional Design Competencies: The Standards (2013), instructional designers should be able to (Koszalka, Russ-Eft, and Reiser 2013, 46) “use an instructional design and development process appropriate for a given project (essential). This is an essential competency and it includes three performance statements, one essential and two advanced: (a) Select or create an instructional design process based on the nature of the project (essential); (b) Modify the instructional design process as project parameters change (advanced); (c) Describe a rationale for the selected, created or modified instructional design process (advanced).” As explained by the Standards, instructional designers must adapt the instructional design model they use to project parameters. To do that, they must be familiar with many models of instructional design. This chapter provides additional information about this competency, providing suggestions for instructional designers on how to demonstrate the competency. The chapter introduces how to use an instructional design and development process appropriate for a project, select or create an instructional design process based on the project’s nature, modify the instructional design process as project parameters change, and describe the rationale for the selected, created, or modified instructional design process.

119

120

Mastering the Instructional Design Process

Traditionally, instructional design has been dominated by the ADDIE model (or some version of it), a hallmark of instructional design and an acronym that stands for analyze, design, develop, implement, and evaluate. But in recent years, instructional designers have grown more eclectic in their viewpoint (Rothwell, Zaballero, Asino, Briskin, Swaggerty and Bienert 2015), preferring to choose a governing model that depends on the project situation. Eclecticism has grown as a direct consequence of greater pressure to deliver faster instructional products and services (Korkmaz and Boling 2014). Additional pressures are created by an increasingly robust range of software and hardware to support instruction that may be used individually or collectively, increasing the need to find flexible, agile, and innovative approaches that will work across many platforms, many possible national cultures, and with many media used alone or in a blend. Consider such practical questions as these: • When does the client wish to have the instructional design project finished? • How much is the client willing to pay for the project? • How rigorous does the client wish the process to be? • What media should be used to design and deliver instructional design projects? • How many languages, media, and technology mixes must be used to complete the project? • How many and what kind of people are available to work on an instructional design team, and how well-equipped are they to meet project requirements? These and similar practical considerations may influence how instructional designers carry out their projects. Time and cost issues usually trump all other considerations—including such critically important issues as the quality of the instructional products, the media that may be used, and learner transfer back to the job. As a direct consequence, instructional designers have grown more willing to draw on many instructional design models and processes by which to carry out their work and get results. While the ADDIE model remains a mainstream approach with which most instructional designers are familiar, experimentation continues using other models perceived to be less cumbersome, more flexible, and faster in getting results. Figure 7.1 summarizes what research indicates are the models that instructional designers use most.

Using an Instructional Design Process Appropriate for a Project

121

FIGURE 7.1. INSTRUCTIONAL DESIGN MODELS: HOW OFTEN DO YOU USE THE FOLLOWING MODELS? ADDIE (Analysis, Design, Development, Implementation, Evaluation) Model

39%

Bloom’s Learning Taxonomy

29%

Kirkpatrick’s Four Levels of Training Evaluation Gagné’s Nine Events of Instruction HPI (Human Performance Improvement) Model SAM (Successive Approximation Model) ROPES (Review/relate, Overview, Presentation, Exercise, Summary) Model Dick and Carey Model

Cathy Moore’s Action Mapping

28%

9%

39%

35%

35%

17%

7% 14% 3% 17% 3% 10% 3% 10% 2% 7% 2%

Merrill’s First Principles of Instruction

6%

Kemp’s Instructional Design Model

2% 7%

Always

Often

Source: W. Rothwell, A. Zaballero, T. Asino, J. Briskin, A. Swaggerty, and S. Bienert, Skills, Trends, and Challenges in Instructional Design (Alexandria, VA: ATD Press, 2015). Used by permission of ATD.

122

Mastering the Instructional Design Process

While not all models used are sufficient alone to guide instructional design without some creative uses, they include in no particular order (see http://ezinearticles.com/?A-Review-of-the-Instructional-Design-Models& id=6913502): • Merrill’s First Principles of Instruction (see: http://web.cortland.edu/ frieda/id/IDtheories/44.html). • Bloom’s Learning Taxonomy (see: ww2.odu.edu/educ/roverbau/ Bloom/blooms_taxonomy.htm). • Kirkpatrick’s Four Levels of Training Evaluation (see: www.kirkpatrick partners.com/OurPhilosophy/TheKirkpatrickModel). • Gagné’s Nine Events of Instruction (see: www.instructionaldesign.org/ theories/conditions-learning.html). • Kemp’s IDM (see: http://idmethodology.blogspot.mx/2014/02/ continued-instructional-design-models_25.html). • Keller’s ARCS Model of Motivational Design (Attention, Relevance, Confidence, Satisfaction) (see: www.learning-theories.com/kellersarcs-model-of-motivational-design.html). • ASSURE Model (Analyze Learners, State Objectives, Select Methods, Media, and Materials, Utilize Media and Materials, Require Learner Participation, and Evaluate and Revise) (see: www.instructionaldesign.org/ models/assure.html). • Smith and Ragan’s IDM (see: www.angelachristopher.net/uploads/8/ 3/2/4/832462/model_resourceassignment.pdf). • Cathy Moore’s Action Mapping (see: http://blog.cathy-moore.com/ action-mapping-a-visual-approach-to-training-design/). • The Successive Approximation Model (SAM) (see: www.alleninteractions .com/sam-process). • Human Performance Improvement (HPI) model (see: theengagement effect.com/our-toolset/additional-tools/human-performanceimprovement). • ROPES (Review/relate, Overview, Presentation, Exercise, Summary) (see: http://business.baylor.edu/James_Moshinskie/ROPES.HTM). Aspiring instructional designers will grow familiar with these models, drawing on them—and others—as clients and project demands require. Most are not incompatible; rather, they can usually complement each other.

Using an Instructional Design Process Appropriate for a Project

123

Selecting or Creating an Instructional Design Process Based on the Project Many instructional design models have been published (see Reigeluth 1999a, 1999b, and 2008; Reigeluth and Carr-Chellman 2009; Richey, Klein, and Tracey 2010). While most published models of instructional design preserve a version of the ADDIE model because they require analysis, design, development, implementation, and evaluation, the race is on to find a model that strikes a four-fold balance in being robust, research- and theory-based, fast, and cost-effective. An important assumption is that no model or process works all the time for every occasion, setting, or learner group. Instructional designers do not assume that all projects will follow the same project flow; rather, instructional designers are increasingly willing to modify their project flow and process to match project requirements and demands. The theoretically puristic, and sometimes linear, views of the instructional design past have yielded to a more practical and holistic view. Summaries of several—but not all—instructional design models can be found online at www.instructionaldesigncentral.com/htm/IDC_ instructionaldesignmodels.htm. Review them but realize that no model or process meets all requirements because one size does not fit all situations, clients, or learners. Familiarity with these different approaches is, however, important because they provide a foundation for selecting an optimum approach to meet instructional design project requirements.

Modifying the Instructional Design Process as Project Parameters Change Instructional design is as much art as science. Science tells us how the brain works and how learning can be based on what is known about brain anatomy and chemistry. It is also the product of years of research on learning (Ambrose, Bridges, DiPietro, and Lovett 2010). But the art comes in when working on projects. Instructional designers must know how to be adaptable and be willing to use “what works.”

124

Mastering the Instructional Design Process

Describing a Rationale for the Selected, Created, or Modified Instructional Design Process Those who are eclectic in their outlooks must be prepared to explain why such eclecticism might be necessary to other instructional designers and to clients such as managers and workers. That must usually be done in project terms. Clients rarely care about the delicacy of theory but do care about the cost, time, and learning effectiveness of project parameters. Instructional design theories often have implications for how an instructional design project is organized and carried out and how instruction itself is organized. It is the project implications, and the impact of the learning, that can justify theoretical eclecticism.

CHAPTER EIGHT

ORGANIZING INSTRUCTIONAL PROGRAMS AND/OR PRODUCTS

T

his chapter addresses the instructional designer’s skills and abilities in organizing instructional programs and/or products. As Koszalka, Russ-Eft, and Reiser (2013) explain, this competency requires instructional designers to “organize instructional programs and/or products to be designed, developed, and evaluated (essential).” Instructional designers should (a) “determine the overall scope of instructional programs and/or products (advanced); (b) identify and sequence instructional goals (essential); and (c) specify and sequence the anticipated learning and performance outcomes (essential).” This chapter addresses these issues.

Determine the Overall Scope of Instructional Programs What should be the scope of instructional programs and/or products? Scope refers to its length and depth. How long should it be? How deeply should it address the topic and attempt to meet the objectives? The answers to these important questions will depend on learner needs and on client preferences—and budget. Keenly felt needs will command time, attention, and management commitment. Those less keenly felt will not command time, attention, commitment—or money.

125

126

Mastering the Instructional Design Process

Instructional designers must explore these questions with their management clients to determine the answers to the scope questions posed above. This is an advanced skill because mere technical ability will not provide an answer. Instructional designers who are proficient at this skill are able to work within the organization’s political system. In short, an underlying issue is really one of power—what is important to an organization and how to exercise power to make it happen. Identify and Sequence Instructional Goals Once work analysis has been performed, instructional designers should be ready to write statements of performance objectives. Sometimes used synonymously with instructional or behavioral objectives, performance objectives are necessary for one very important reason: they guide remaining steps in the instructional design process by describing precisely what the targeted learners should know, do, or feel on completion of a planned learning experience. It is also highly desirable if they can also communicate the on-the-job results sought from the learning experience. In a sense, performance objectives create a vision of what learners should be doing after they master the instruction. Objectives focus on the results of instruction, what learners should know, do, or feel upon completion of a learning experience (Mager 1997b). Objectives do not focus on what instructional designers should do or what activities trainers should use to effect changes in learners’ work performance. Of course, as with many other topics in the instructional design field in recent years, some critics have weighed the pros and cons of objectives, questioning what they are for and why. Distinguishing Performance Objectives from Goals and Activities. Performance objectives should not be confused with goals or activities. But what are instructional or organizational goals? What are learner or trainer activities? How do they differ from performance objectives? Let us begin the chapter by answering these questions. What Are Instructional and Organizational Goals?

Instructional goals are simply expressions of the general results desired from instruction. Unlike performance objectives, they are not measurable. In a famous explanation, Mager (1997a) calls them warm fuzzies because

Organizing Instructional Programs and/or Products

127

they sound desirable (warm) but are so vague (fuzzy) that achieving them is unclear. In fact, different people may assign their own meanings or significance to them. Examples of instructional goals are easy enough to point out and include such lofty efforts as “improving customer service,” “improving quality,” “increasing profitability,” and “increasing learner understanding.” However, they are warm fuzzies, as Mager uses that term, because they do not clarify precisely what a learner must do or how a learner should perform. Organizational goals are results desired from an organization. Often included as part of an organization’s formal mission statement, they articulate philosophy, embody management values, and imply an organization’s general direction. Their achievement is rarely restricted to a specific time. Nor do goals lend themselves, as organizational objectives do, to specific measurement methods. As expressions about an organization, their link to individual job performance is often unclear. Examples of organizational goals include “serving the community,” “maintaining a safe and productive workplace for employees,” and “making a reasonable return on investment.” To become measurable, organizational goals must be translated into organizational objectives. To focus on individuals, organizational goals must be translated into terms that are directly related to what employees do and how well they do it. What Are Learner-Trainer Activities? A learner activity refers to what a learner is doing during a planned learning experience. For example, “listening to a lecture”—admittedly a passive activity because it implies more action by a trainer than by a learner—is a learner activity. Another example is “answering the questions at the end of a case study.” Activities emphasize behaviors; in contrast, performance objectives emphasize results. A trainer activity refers to what a trainer is doing during a planned learning experience. For instance, one trainer activity is “defining terms.” Other examples include lecturing, introducing a learning activity, showing a videotape, or passing out evaluations. Trainers sometimes focus on what they should do during a learning experience rather than on what learners can do by the end of instruction. How Do Performance Objectives Differ from Goals and Activities? A performance objective is an expression of a desired result of a learning experience. It differs from a performance goal in that it is measurable and is an expression of what should be achieved. It differs from activities in that it describes desired results, not behaviors leading to results.

128

Mastering the Instructional Design Process

Deriving Performance Objectives from Goal Analysis and Task or Content Analysis. Instructional designers can derive performance objectives from

goal analysis, carried out with instructional and organizational goals and learner-trainer activities, or from task or content analysis results. But what is goal analysis, and how is it carried out? How are the results of task and content analysis used to write performance objectives? Let us turn to these questions next. Defining Goal Analysis. Goal analysis is a means of transforming laudable

but otherwise vague desires into specific and measurable targets for learner accomplishment (Mager 1997a). Goal analysis is appropriate to use on those many occasions when instructional designers are approached by their clients to work miracles. Clients often speak in terms of vague and ill-defined goals, and instructional designers must use methods such as performance analysis to decide what kind of performance problem exists. Goal analysis is a later step, intended to determine precisely what measurable results are desired from an instructional design solution. Performing Goal Analysis. To perform goal analysis, instructional designers

should carry out five simple steps: 1. Identify the goal, the warm fuzzy, and write it down. Clarify the vague goal that instruction intends to achieve. 2. Write down examples of what people are saying or doing when they are behaving in a way corresponding to the goal. In short, identify behaviors associated with the goal. 3. Sort out unrelated items and polish the list developed in Step 2. Eliminate duplications not clearly associated with achieving the goal. 4. Describe precisely what learners should be doing to demonstrate goal achievement. Statements of this kind become performance objectives. 5. Test the performance objectives to ensure that they are linked to the goal and, when enacted, will lead to the desired instructional results. These five steps can help convert otherwise vague instructional or organizational goals—or learner or trainer activities—into precise and measurable performance objectives. A simple description of the process should clarify it. Suppose a team of instructional designers has been assigned the daunting task of “improving customer service.” (Clients sometimes speak vaguely when they identify perceptions of learner needs.)

Organizing Instructional Programs and/or Products

129

First, the team members would have to make sure the aim is improving customer service, not some other goal. They would do that by analyzing the performance problem and assessing learner needs. Second—assuming a justifiable instructional need was identified—the team members would list specific employee behaviors associated with effective customer service. They would ask these questions: What will people be doing when they are serving customers effectively? What will they be saying? Examples of appropriate behaviors might include answering customer phone calls quickly and courteously, approaching customers politely when they arrive in a store to look at merchandise, and identifying customers’ problems or needs quickly and accurately. (These are just a few examples of behaviors associated with the goal.) Note that even these behaviors can be made more specific if the instructional designers described precisely what an employee does to “act courteously” or “identify customers’ problems.” And the examples just given could easily swell if the team members applied various methods of creative problem solving to identify more behaviors and worker statements associated with “improved customer service.” Once the previous steps have been completed, the instructional designers should then eliminate duplicative behaviors from the list. Finally, team members would write performance objectives and try them out to see whether learners who achieved them would indeed demonstrate “improved customer service” as defined by the clients. Converting Results of Task or Content Analysis into Performance Objectives.

Goal analysis is just one of two primary methods used to identify the specific results desired from instruction. The second, and perhaps more commonly used, method is conversion of task or content analysis results into performance objectives. The results of task analysis reveal how work is, or should be, performed. As we have seen, the results of content analysis also create a logical organizational scheme for subject matter that can be used as a starting point for developing instruction. But there is quite a difference between doing the work—or organizing subject matter—and engineering instruction that will produce learners who can do the work or demonstrate the desired knowledge. For this reason, it is not enough just to analyze how the work is done or how subject matter can be logically organized. Some consideration must also be given to the related, but different, issue of how to produce the desired results of instruction.

130

Mastering the Instructional Design Process

Instructional designers convert the results of task or content analysis into specific performance objectives by 1. Establishing instructional purpose. 2. Classifying learning tasks. 3. Analyzing learning tasks. These steps are depicted in Figure 8.1. First, instructional designers should establish purpose. Purpose means the primary reason for a planned instructional experience. There are typically four choices: (1) increasing learners’ knowledge, (2) changing attitudes or feelings, (3) building skills, or (4) combining one or more of the other three choices. FIGURE 8.1. STEPS FOR CONVERTING RESULTS OF TASK OR CONTENT ANALYSIS INTO PERFORMANCE OBJECTIVES Results of task analysis

Performance objectives

What should workers know or do to perform competently?

What should workers know or do by the end of instruction, how can their performance be judged, and what assumptions must be made about conditions for performance?

Step 1 Establish instructional purpose

Step 2 Classify learning tasks

Step 3 Analyze learning tasks Source: Taken from W. Rothwell and H. Kazanas, Mastering the Instructional Design Process: A Systematic Approach (4th ed.) (San Francisco: Pfeiffer, 2008), 175.

Organizing Instructional Programs and/or Products

131

Second, instructional designers should classify learning tasks by examining each work task and asking this question: What kind of instruction will be necessary to instruct people to perform this task or demonstrate this knowledge? Only four answers to this question are possible. Instruction can be designed for (1) knowledge, (2) feelings, (3) skills, or (4) some combination of the first three. Here are a few examples of ways to classify tasks: Work Task

Classification of Learning Tasks

Explaining a procedure to others Serving customers courteously Typing letters

Knowledge Feelings Skills

Instructional designers should bear in mind that the appropriate way to carry out the instructional design process depends on the results to be achieved. Classifying work tasks into learning tasks is important because it can suggest the best ways to design instruction that is intended to bring about particular results. Of course, more than one classification scheme for work or for learning tasks or content has been devised. For example, in a classic treatment, Gagné, Briggs, and Wager (1992) distinguish among intellectual skills, cognitive skills, verbal information, motor skills, and attitude. As they define them, intellectual skills are equated with the ability to read, write, and compute, as well as capabilities needed to perform tasks in special occupational fields. Cognitive skills underlie learning how to learn, that is, knowing how to get to the heart of problems. Verbal information is linked to summarizing or stating a principle. “You can usually spot a verbal information goal by the verb that is used,” explain Dick and Carey (1990, 34). “Often the learner must state, list, or describe something.” Motor skill is associated with body movement of any kind, ranging from moving a pen to using a computer keyboard. Attitude means a persistent set of beliefs. Since each of these learning tasks is intended to evoke a different result, each calls for different instructional strategies. The third step is to analyze learning tasks, a process called learning task analysis (Gagné, Briggs, and Wager 1992). Not to be confused with work task analysis, its purpose is to identify prerequisite knowledge. A prerequisite describes what learners should know before participating in instruction. Instructional designers use three methods to identify prerequisites: (1) learning hierarchies, (2) cluster analysis, and (3) procedural analysis (Dick and Carey 1990).

132

Mastering the Instructional Design Process

Instructional designers develop a learning hierarchy by repeatedly asking this question of each work task and subtask: What does a learner need to know to do that? (Davis, Alexander, and Yelon 1974). This process is called hierarchical analysis. To cite a simple example: to fill an automobile’s tank with gasoline, a learner must first know what an automobile is, what a gasoline tank is, where the gasoline tank is located, where gasoline may be purchased, how to remove the gas cap, and so on. Each task implies a learning hierarchy. Hierarchical analysis is applied to intellectual, psychomotor, and attitudinal skills—but not to verbal information. To perform hierarchical analysis, instructional designers should simply flowchart the relationship between the work task and the required prerequisite knowledge. They then develop performance objectives from the hierarchy. Cluster analysis is appropriately used with verbal information or attitudes. It is particularly useful in developing performance objectives from results of content analysis and is based on categories of information. For instance, categories might include the number of letters in the alphabet (a fact) or the number of component parts in a social theory. (A social theory is a principle based on opinions of various experts.) To perform a cluster analysis, instructional designers begin by drawing a chart. They place an instructional goal at the top. They then list below it “the major categories of information that are implied by the goal” (Dick and Carey 1990, 58). Instructional designers should try to be creative as they categorize information, but they should remember that one aim is to be as complete as possible. They must succeed in developing a scheme to organize the information. Quite often this process can economize the instructional effort. Procedural analysis is the process of identifying what learners should know to perform one task or a series of related tasks (a procedure). It is appropriately applied to developing performance objectives for intellectual skills, motor skills, and attitudes. But it does not work with verbal information, for which no “step-by-step list of activities” can be created. To perform procedural analysis, instructional designers should first identify an instructional goal and then flowchart steps in the procedure. For each step (task) in the procedure, they should answer this question: What must the learner know, do, or feel to perform? They should then express the answer by stating precise performance objectives. Linking Work Activities and Performance Objectives. Performance objectives must always be tied to work activities. However, they may be linked to different expressions of work activities, for instance, as work tasks are

Organizing Instructional Programs and/or Products

133

presently performed or as they could be more efficiently and effectively performed at present or in the future. Performance objectives can also be linked to subject matter as related to job performance. When learners achieve performance objectives by the end of a planned learning experience, they should be able to perform in the application environment, or at least be familiar with the verbal information on which work performance depends. Instructional designers generally direct their attention to demonstrating learner change by the end of instruction, not on the learner’s return to the application environment, though that view has been changing. To demonstrate achievement of performance objectives in the application environment rather than merely at the end of instruction, they would probably have to devise more than one type of performance objective. Indeed, Briggs (1977) has identified four types of performance objectives. Each reflects a different time span. But instructional designers have seldom expressed performance objectives in terms of on-the-job changes; rather, the traditional focus has been on end-of-instruction changes. On-the-job change requires instructional designers to consider more than just what learners will be able to do: it also requires consideration of what the organization and the learners’ supervisors must do to support the learners’ application of knowledge, skills, and attitudes. Stating Objectives in Performance Terms. Instructional designers should describe the desired results of instruction in performance-based terms. They should be able to classify the type of performance objectives that must be written and then state performance objectives that are directly or indirectly linked to work requirements. The objectives should thus clarify, in measurable terms, what learners should be able to do at the end of instruction, how well they should be able to do it, and what conditions have to exist or equipment must be available for them to exhibit the performance. To write performance objectives, however, instructional designers must have a task or concept analysis and a learner analysis. Classifying Performance Objectives. Instructional designers begin the process of stating performance objectives by identifying the kinds of objectives that must be written. Referring to the task classification prepared earlier in the instructional design process, they should clarify whether each objective will focus on knowledge, skills, or attitudes. The most commonly used classification scheme for performance objectives was first described in 1956. That year, Bloom and his colleagues

134

Mastering the Instructional Design Process

published the Taxonomy of Educational Objectives and defined three domains of learning—knowledge, attitudes, and skills. Objectives focused on increasing learner knowledge are called cognitive objectives; objectives focused on changing learners’ attitudes are called affective objectives; and objectives focused on building skills are called psychomotor objectives. Knowledge means “facts and information essential to performing a job or task,” skills involve the “ability to behave in ways associated with successful job performance,” and attitudes are “feelings about performance that are voiced to other people.” Each “domain” of learning consists of increasingly complicated levels, as shown in Figures 8.2, 8.3, and 8.4. Instructional designers begin classifying performance objectives by identifying the level of the domain that they are trying to reach. When they have done that for the end results desired from a planned learning experience, they are ready to begin writing performance objectives.

FIGURE 8.2. LEVELS OF OBJECTIVES IN THE COGNITIVE DOMAIN Highest Level Evaluation

Assessing the value of ideas, things, and so on

Synthesis

Assembling a whole from parts

Analysis

Disassembling a whole into parts

Application

Using what has been previously learned

Comprehension

Knowing what a message means

Knowledge

Remembering/recalling terms, facts, and so on

Lowest Level Source: Taken from W. Rothwell and H. Kazanas, Mastering the Instructional Design Process: A Systematic Approach (4th ed.) (San Francisco: Pfeiffer, 2008), 178.

Organizing Instructional Programs and/or Products

135

FIGURE 8.3. LEVELS OF OBJECTIVES IN THE AFFECTIVE DOMAIN Highest Level Characterization

Adopting a new way of life or outlook

Organization

Developing/acquiring a new value system

Valuing

Accepting values/beliefs

Responding

Participating

Receiving

Paying attention

Lowest Level

Source: Taken from W. Rothwell and H. Kazanas, Mastering the Instructional Design Process: A Systematic Approach (4th ed.) (San Francisco: Pfeiffer, 2008), 179.

Describing Parts of Performance Objectives. Performance objectives make tangible a vision of what learners should know, do, or feel at the end of a planned instructional experience. They should contain statements about at least two of the following three components (Mager 1988):

1. Performance 2. Criterion 3. Condition Other components could also be included in performance objectives. They may include targeted participants (Mager 1988). They may even include a description of how instruction will be carried out (Mager 1988). The performance component of an objective describes how a learner will demonstrate proficiency. It is an activity or behavior to be learned during instruction and demonstrated afterward. A statement of performance

136

Mastering the Instructional Design Process

FIGURE 8.4. LEVELS OF OBJECTIVES IN THE PSYCHOMOTOR DOMAIN Highest Level Complex Overt Response

Performing automatically with facility/habitually

Mechanism

Acting without assistance

Guided Response

Performing a task with assistance

Set

Getting ready to perform

Perception

Observing behaviors involved in a task

Lowest Level

Source: Taken from W. Rothwell and H. Kazanas, Mastering the Instructional Design Process: A Systematic Approach (4th ed.) (San Francisco: Pfeiffer, 2008), 180.

always begins with a verb, and the choice of verb is typically linked to the type of task to be learned. The criterion component of an objective describes, in measurable terms, just how well participants must perform to demonstrate competence (Mager 1988). It is worth emphasizing that a criterion must be measurable. Measures may be expressed by quantity, quality, cost, time, or customer requirements. Criteria may be derived from past work practices, present performance needs, future organizational plans, academic and government research, customer-focused research, benchmarking with best-in-class organizations, and other sources. They should be tied to historical performance standards or future organizational plans. There are two different kinds of criteria: process and product (Blank 1982). A process criterion describes how well the learner should perform the task; a product criterion describes the product of the task. Examples of

Organizing Instructional Programs and/or Products

137

process criteria include “following company procedures,” “conforming to the organization’s safety practices,” and “within ten minutes.” Product criteria include any of the following: “to the client’s satisfaction,” “with fewer than three errors,” and “ready for sale.” To establish criteria, instructional designers should ask questions such as the following: • How has competent work performance historically been identified through measurable means? • How well must the work task associated with this performance objective be performed for the organization to meet its present competitive needs? • How can these organizational needs be expressed as measurable results to be achieved? • How well must the work task be performed in the future to help the organization achieve its strategic business plans? • How can performance be measured? • How will the consequences of task performance be measured? In some cases, there will be a difference between existing and possible criteria. In other words, workers can simply perform better than they have been performing. One way that instructional designers can gauge the possibility for productivity improvement is to subtract the difference between the work output of the highest and lowest performers. It too can serve as the basis for criteria in performance objectives. The condition component of a performance objective explains what working conditions must exist when the performer demonstrates his or her knowledge, skill, or ability (Mager 1988). Conditions may include essential or desirable situations in which performance is necessary. Condition statements usually begin with the word “given,” as in the following phrase: “given a ruler, the learner will be able to measure inches.” In this context, “given” means “the learner is provided with some equipment, resources, or information with which to function and cannot perform competently without them.” Writing Performance Objectives. To write performance objectives, instruc-

tional designers should begin with the following sentence or some variation of it: “On completion of instruction, learners should be able to . . . .” They should then list the performance objectives, beginning each phrase with a verb. The portion of the objective that begins with the verb is the

138

Mastering the Instructional Design Process

performance component. It is usually followed by statements about criterion and condition. Of course, criterion addresses this question: How well should the performance be done? It should always be measurable. The condition component addresses the following question: What equipment or other resources are necessary for the performance to be demonstrated by the learner? Avoiding Common Mistakes in Writing Performance Objectives. Writing per-

formance objectives is more difficult than it may appear at first blush. Some mistakes are relatively common. They are worth describing so they can be avoided. 1. Avoid making objectives long-winded. Try to make them as concise as possible. 2. Do not use vague language. Words and phrases such as “understand,” “demonstrate familiarity with,” or “know” should usually be avoided because they are vague. 3. Try to avoid descriptions of criteria that are linked to instructor (or supervisor) satisfaction, as in the phrase “will perform to the satisfaction of the instructor.” The reason: performance objectives of this kind lead to arbitrary differences in assessments of learner achievement. 4. Avoid lengthy “laundry lists” of required equipment and other resources when describing the conditions necessary for performance. List only the equipment and other resources that would not be obvious to a reasonable person.

Specify and Sequence the Anticipated Learning and Performance Outcomes Sequencing was discussed in Chapter Five. The same approaches used to sequence skills and knowledge can be used to specify and sequence the anticipated learning and performance outcomes. There is little need to repeat the approaches. It is important, however, to remember to place performance objectives in the appropriate order for learning. Refer to Figure 8.5 for some rules on sequencing performance objectives.

Organizing Instructional Programs and/or Products

139

FIGURE 8.5. RULES FOR SEQUENCE PERFORMANCE OBJECTIVES: A FLOWCHART START Prerequisite Information Are you ready to begin sequencing performance objectives, having completed • Draft statements of performance NO objectives? • Statements of learner/trainee characteristics? • Analysis of job, task, or content? • A needs assessment/analysis? • A setting analysis?

Prepare to sequence performance objectives by gathering necessary prerequisite information.

YES

Learning Task Can the entire learning task (each terminal objective) be classified as an effort to communicate verbal NO information, develop intellectual skills, help learners craft their own cognitive strategy, build motor skills, or shape or change learners’ attitudes?

Prepare enabling objectives for each terminal objective and then classify each enabling objective.

YES

Appropriate Sequencing Procedure Has the appropriate sequencing procedure for each learning task been considered? YES

NO

Review appropriate sequencing procedure for performance objectives intended to communicate verbal information, develop intellectual skills, help learners craft their own cognitive strategy, build motor skills, or shape or change learners’ attitudes.

140

Mastering the Instructional Design Process

FIGURE 8.5. RULES FOR SEQUENCE PERFORMANCE OBJECTIVES: A FLOWCHART, cont’d.

Reality Check Is there good reason to consider other matters in making a decision NO about sequencing performance objectives?

Sequence terminal and/or enabling performance objectives based on the learning task.

YES

Learners Is it clear what learners already know and/or can do?

NO

YES

Learners' Attitudes Are the learners motivated to learn for any of the following reasons • They are experiencing a “teachable moment”? • They can readily see the importance of achieving the NO performance objectives for themselves and/or the organization? • They believe the performance objectives are attainable? • They favor—or are at least neutral to—change? • They believe they will be rewarded, directly or indirectly, for achieving the results expressed in performance objectives? YES

Clarify the targeted learners and assess their existing knowledge/ skills. If instruction is geared to novices and the learning task warrants it, consider prerequisite knowledge or skill. Check to ensure that the learners possess that prerequisite knowledge or skill. If they do not, backtrack to ensure that prerequisites are met before learners try to achieve these performance objectives.

Consider methods of sequencing performance objectives so as to motivate the learners. Do that by demonstrating, before anything else, the importance of the instruction to them and to the organization.

Organizing Instructional Programs and/or Products

141

FIGURE 8.5. RULES FOR SEQUENCE PERFORMANCE OBJECTIVES: A FLOWCHART, cont’d.

Learners' Individual Learning Styles Has some effort been made to sequence performance objectives based on individual learning style, giving learners some flexibility in how they learn?

NO

YES Reality Check Is there good reason to consider other matters in making a decision about sequencing performance objectives?

NO

YES Learning Environments Is it clear how many learning environments affect the ability of learners to achieve the performance objectives at present and in the future?

NO

Make the effort to sequence performance objectives in ways that will appeal to various styles of learning. If time permits, do up-front assessment of the targeted learners to determine if one type of learning style predominates within the group. If so, use that information when making decisions about sequencing performance objectives.

Sequence terminal and/or enabling performance objectives based on the learning task and/or learner characteristics.

Clarify how many learning environments affect the ability of learners to achieve the performance objectives.

YES The Environment Outside the Organization Is it clear • What changes or trends outside the organization are occurring at present or will probably occur NO in the future? • How those changes may affect the knowledge or skills of workers? • How those changes should affect the sequencing of performance objectives? YES

Clarify • What changes or trends outside the organization are occurring at present or will probably occur in the future. • How those changes may affect the knowledge or skills of workers. • How those changes should affect the sequencing of performance objectives.

142

Mastering the Instructional Design Process

FIGURE 8.5. RULES FOR SEQUENCE PERFORMANCE OBJECTIVES: A FLOWCHART, cont’d.

The Organization's Environment/Culture Is there support for learners to achieve performance objectives among the organization’s managers, union officials, and workers?

NO

Create an impetus for change—or risk ineffective instruction that will not transfer from the learning environment to the work environment.

YES

The Learning Situation Is it clear what problems, changes, or other job-related issues will probably concern learners at the time instruction is offered to them?

NO

YES

The Learning Situation Have you taken into account problems or job-related issues that may subsequently affect learners’ willingness to apply what they learned on their jobs?

NO

YES

Reality Check Is there good reason to consider your own experience—or that of other instructional designers—in making a decision about sequencing performance objectives? YES

NO

Clarify what other problems, changes, or other job-related issues will probably concern learners at the time instruction is offered to them and use that information to sequence performance objectives so that they help learners cope with practical problems.

Take into account problems or job-related issues that may subsequently affect learners’ willingness to apply what they learned on their jobs as you make decisions about sequencing terminal and/or enabling performance objectives.

Make the decision about sequencing performance objectives, considering various alternative methods of sequencing.

Organizing Instructional Programs and/or Products

143

FIGURE 8.5. RULES FOR SEQUENCE PERFORMANCE OBJECTIVES: A FLOWCHART, cont’d.

Your Instructional Design Experience Is there support for learners to achieve performance objectives among the organization’s managers, union officials, and workers?

NO

Consult your own previous experience with similar types of instruction before making a decision about sequencing terminal and/or enabling performance objectives.

YES

Others' Instructional Design Experience Have you consulted other instructional designers about their experience with similar types of instruction before making a decision about sequencing terminal and/or enabling performance objectives?

NO

Consult others about effective methods of sequencing terminal and/or enabling performance objectives for similar types of instruction.

YES

Reality Check Should no single issue—such as desired learning outcomes, the targeted learners, the learning environment, or previous instructional design experience—be given priority in making the decision about sequencing terminal and/or enabling performance objectives?

NO

Identify the priority issue(s) governing the sequencing decision and delimit the choices to the most appropriate one.

YES

Reality Check Have you identified the range of sequencing choices that are applicable in this situation? YES

NO

Identify the range of sequencing choices applicable to the situation while eliminating any choices that are clearly not appropriate.

144

Mastering the Instructional Design Process

FIGURE 8.5. RULES FOR SEQUENCE PERFORMANCE OBJECTIVES: A FLOWCHART, cont’d.

Considering Alternative Sequencing Methods Have you made an effort to consider, briefly, how each sequencing choice might be applied to this situation?

NO

Consider how each sequencing choice might be applied to this situation.

YES

Making the Sequencing Decision Make the decision how to sequence terminal and/or enabling performance objectives, choosing from such alternatives as • Chronological • Topical • Whole-to-part • Part-to-whole • Known-to-unknown • Unknown-to-known • Step-by-step • Part-to-part-to-part • General-to-specific

Prepare and Test Instructional Materials Based on the Sequencing Decision Having made the sequencing decision, select an instructional strategy, prepare an outline for instruction, source instructional materials, and test the materials. Source: Taken from W. Rothwell and H. Kazanas, Mastering the Instructional Design Process: A Systematic Approach (4th ed.) (San Francisco: Pfeiffer, 2008), 217–222.

CHAPTER NINE

DESIGNING INSTRUCTIONAL INTERVENTIONS

A

ccording to Instructional Design Competencies: The Standards (2013), instructional designers should be able to “design instructional interventions (essential). This is an essential competency and it includes six performance statements, all of which are essential: (1) identify instructional strategies that align with instructional goals and anticipated learning outcomes (essential); (2) apply appropriate interaction design and interactive learning principles (essential); (3) Use appropriate message and visual design principles (essential); (4) Apply appropriate motivational design principles (essential); (5) Accommodate social, cultural, political, and other individual factors that may influence learning (essential); (6) Select appropriate technology and media to enhance instructional interventions, taking into account theory, research, and practical factors (essential)” (Koszalka, Russ-Eft, and Reiser 2013, 48). This chapter provides additional information about this competency, noting how instructional designers should identify instructional strategies, use message and visual design principles, apply appropriate motivational design principles, and accommodate factors—social, cultural, and political—that will influence learning. Selecting technology and media to enhance instructional interventions is addressed in another chapter. Having written and sequenced performance objectives, instructional designers are ready to plan instruction. They should begin by asking this question: How can the desired results of instruction be achieved? The answer is through instructional strategy. (See Table 9.1.) While instructional strategy 145

146

Mastering the Instructional Design Process

TABLE 9.1. INSTRUCTIONAL STRATEGIES AND TACTICS 1. Contextualizing instructiona 1.1 Gaining the attention of the learner 1.1.1 arouse learner with novelty, uncertainty, surprise 1.1.2 pose question to learner 1.1.3 learner poses question to be answered by lesson 1.2 Relate the goals of instruction to the learner’s needs 1.2.1 explain purpose or relevance of content 1.2.2 present goals for learners to select 1.2.3 ask learners to select own goals 1.2.4 have learner pose questions to answer 1.3 State the outcomes of instruction 1.3.1 describe required performance 1.3.2 describe criteria for standard performance 1.3.3 learner establishes criteria for standard performance 1.4 Present advance organizers 1.4.1 1.4.2 1.4.3 1.4.4 1.4.5

verbal expository: establish context for content verbal comparative: relate to content familiar to learner oral expository: establish context for instruction oral comparative: relate to content familiar to learner pictorial: show maps, globes, pictures, tables

1.5 Present structured overviews and organizers 1.5.1 outlines of content: verbal (see also 1.4.1, 1.4.2) 1.5.2 outlines of content: oral (see also 1.4.3, 1.4.4) 1.5.3 graphic organizers/overviews 1.5.4 combinations of verbal, oral, and pictorial overviews 1.6 Adapt context of instruction 1.6.1 content adapted to learner preferences (different situations) 1.6.2 content adapted to prior knowledge 2. Present and cue lesson content 2.1 Vary lesson unit size 2.1.1 large chunks 2.1.2 small chunks 2.2 Present vocabulary 2.2.1 2.2.2 2.2.3 2.2.4 2.2.5 2.2.6 a Key

present new terms plus definitions student looks up list of new terms present attributes of rule, definition, concept, principle paraphrase definitions, present synonyms present definitions derive definitions from synonym list

steps of instruction are in bold print; tactics are underlined.

Designing Instructional Interventions

147

TABLE 9.1. INSTRUCTIONAL STRATEGIES AND TACTICS, cont’d. 2.3 Provide examples 2.3.1 prototypical examples 2.3.2 matched example/non-example pairs 2.3.3 divergent examples 2.3.4 close-in non-examples 2.3.5 vary the number of examples 2.3.6 model appropriate behavior 2.4 Use cuing systems 2.4.1 provide graphic cues: lines, colors, boxes, arrows, highlighting 2.4.2 provide oral cues: oral direction 2.4.3 provide auditory cues: stimulus change (e.g., music, sound effects, voice change) 2.4.4 provide type style cues: font changes, uppercase, type size, headings, hierarchical numbering system, indentation 2.4.5 present special information in windows 2.5 Advise learner 2.5.1 instructional support needed: number of examples, number of practice items, tools, materials, resources 2.5.2 learning strategies to use 3. Activating learner processing of instruction 3.1 Elicit learner activities 3.1.1 review prerequisite skills or knowledge 3.1.2 learner selects information sources 3.1.3 learner selects study methods 3.1.4 learner estimates task difficulty and time 3.1.5 learner monitors comprehension 3.1.6 learner relates questions to objectives 3.1.7 learner recalls elaborations 3.1.8 learner evaluates meaningfulness of information 3.2 Elicit recall strategies 3.2.1 underline relevant material 3.2.2 rehearse/repeat/re-read 3.2.3 use mnemonic strategies 3.2.4 close reading activities 3.2.5 identification with location (loci method) 3.2.6 create summaries: hierarchical titles 3.2.7 create summaries: prose 3.2.8 create summaries: diagrammatic/symbolic (math) 3.2.9 create summaries: mind maps 3.3 Facilitate learner elaborations 3.3.1 imaging (creating images) 3.3.2 inferring from information 3.3.3 generating analogies 3.3.4 creating story lines: narrative description of information

148

Mastering the Instructional Design Process

TABLE 9.1. INSTRUCTIONAL STRATEGIES AND TACTICS, cont’d. 3.4 Help learners integrate new knowledge 3.4.1 paraphrase content 3.4.2 use metaphors and learner-generated metaphors 3.4.3 generating examples 3.4.4 note-taking 3.5 Help learners organize information 3.5.1 analysis of key ideas 3.5.2 create content outline 3.5.3 categorize elements 3.5.4 pattern note techniques 3.5.5 construct concept map 3.5.6 construct graphic organizers 4. Assessing learning 4.1 Provide feedback after practice 4.1.1 confirmatory, knowledge of correct response 4.1.2 corrective and remedial 4.1.3 informative feedback 4.1.4 analytical feedback 4.1.5 enrichment feedback 4.1.6 self-generated feedback 4.2 Provide practice 4.2.1 massed practice session 4.2.2 distributed practice session 4.2.3 over-learning 4.2.4 apply in real world or simulated situation (near transfer) 4.2.5 change context or circumstances (far transfer) 4.2.6 vary the number of practice items 4.3 Testing learning 4.3.1 pretest for prior knowledge 4.3.2 pretest for prerequisite knowledge or skills 4.3.3 pretest for endpoint knowledge or skills 4.3.4 embedded questions throughout instruction 4.3.5 objective referenced performance 4.3.6 normative referenced performance 4.3.7 incidental information, not objective referenced 5. Sequencing instructional events 5.1 Sequence instruction in logical order 5.1.1 deductive sequence 5.1.2 inductive sequence 5.1.3 inductive sequence with practice 5.2 Sequence instruction in learning prerequisite order 5.2.1 hierarchical, prerequisite sequence 5.2.2 easy-to-difficult 5.2.3 concrete-to-abstract

Designing Instructional Interventions

149

TABLE 9.1. INSTRUCTIONAL STRATEGIES AND TACTICS, cont’d. 5.3 Sequence instruction in procedural order 5.3.1 procedural, job sequence 5.3.2 information processing sequence (path sequencing) 5.3.3 algorithmic presentation 5.3.4 procedural elaboration 5.4 Sequence instruction according to content organization 5.4.1 general-to-detailed (progressive differentiation) 5.4.2 conceptual elaboration 5.4.3 theoretical elaboration 5.5 Sequence instruction according to story structure 5.5.1 narrative sequence Source: Taken from W. Rothwell and H. Kazanas, Mastering the Instructional Design Process: A Systematic Approach (4th ed.) (San Francisco: Pfeiffer, 2008), 241–244.

sometimes refers to the methods, techniques, and devices for instructing, the term is used here to mean strategies for instructing others how to go about the instructional process (see Chen and Bryer 2012; Gregory and Chapman 2012; Moore 2014). Methods, techniques, and devices for instructing are described in the next chapters. In this chapter, we will define instructional strategy. We will also distinguish between two kinds of strategy (macroinstructional and microinstructional strategy) and between instructional strategy and instructional tactics. In addition, we will describe how to conceptualize instructional strategy, choose strategy and tactics, choose media and delivery methods, and judge and justify strategy once chosen. We also summarize cognitive strategy. We also identify ethical and cross-cultural issues in specifying instructional strategies. Great care should be taken in selecting the media—the approach used to deliver instruction. While much attention has focused on using e-learning and blended learning in recent years, there is still room for less technologically dependent methods.

Identifying Instructional Strategies Aligned with Instructional Goals and Anticipated Learning Outcomes In the most general sense, an instructional strategy is perhaps best understood as an overall plan governing instructional content (What will be

150

Mastering the Instructional Design Process

taught?) and process (How will it be taught?). Any strategy is an effort intended to achieve a goal (a result) using various methods. Military strategists attempt to win a war or battle; business strategists seek competitive advantage; instructional strategists try to achieve an instructional goal. An instructional strategy is no different. It is a plan for systematically exposing learners to experiences that will help them acquire verbal information, establish cognitive strategy, or develop intellectual skills, motor skills, or new attitudes. Instructional strategy should grow out of an analysis of the work tasks that learners are being instructed to perform and from the corresponding performance objectives established to achieve those desired results. An instructional strategy demonstrates what to do to achieve results. Once decided on, an instructional strategy helps to select or prepare instructional materials and provides a way to evaluate materials. Instructional strategy should usually be described in writing, although not in the detail typical of final instructional materials comprising a module, unit, or lesson. It should be prepared before instructional materials are designed or selected from other sources. The aim of establishing an instructional strategy is to plan holistically. It helps instructional designers conceptualize, before they begin time-consuming and expensive preparation or selection of instructional materials, what must be done to facilitate learning. In planning instructional strategy, instructional designers should take care to match the method with the objectives. They should also avoid the tendency—too often evident—to seize on using emerging instructional technologies for their own sake. Discussions in recent years have focused around the need to ensure interactivity, variously defined, when using e-learning. Of greatest interest at the moment is so-called blended learning, which makes use of several media in a “blend.” But ensuring that blend calls for skill in using each medium. Distinguishing between Two Kinds of Instructional Strategy There are two kinds of instructional strategy. A macro-instructional strategy is, on the one hand, an overall plan governing a discrete learning experience, such as a course or module. It is the way instructional designers plan to help learners achieve terminal performance objectives. Think of it as a big-picture road map for an entire planned learning experience, akin to a syllabus for a college course. A micro-instructional strategy is a plan governing each part of the learning experience, such as a unit or lesson within a

Designing Instructional Interventions

151

course or module. It is the way instructional designers plan to help learners achieve enabling objectives. An example would be an outline for one planned learning experience or course session. Macro-instructional strategy should typically be specified first. Distinguishing between Instructional Strategy and Instructional Tactics An instructional tactic is related to, but more specific than, an instructional strategy. It is any instructional activity undertaken to facilitate a strategy. All grand instructional strategies are enacted through simple instructional tactics, just as any long-term strategy for winning a game is realized through many short-term plays or activities. Conceptualizing Instructional Strategy There are two ways to think about instructional strategy. The first stems from the philosophy of the instructional designer about learning and instructing; the second stems from events of instruction and conditions of learning. Each way provides guidance when instructional designers find it necessary to identify the range of available instructional strategies. Instructional Strategy Based on Philosophy of Learning and Instructing Instructional designers have fought contentious battles, stemming from contrasting philosophical views, about learning and instructing. Two theoretical positions about learning and instructing seem to represent major anchor points on a philosophical continuum of instructional strategies, according to a classic description by Romiszowski (1981). Some instructional designers believe that all learning can be described best as resulting from a process of reception. This view leads to expositive instructional strategies. But other instructional designers believe that all learning is best described as resulting from a process of learner discovery. This view leads to experiential instructional strategies. To the behaviorist adherents of reception learning, learning centers on the communication process. Learning occurs through exposure to environmental variables outside the learner; instructing means manipulating those variables to achieve predetermined ends. Learning occurs as people receive, understand, apply, and act on information directed to them by others. Learners are passive recipients of instructional messages, instructors or instructional designers are active transmitters of those messages,

152

Mastering the Instructional Design Process

and instruction itself is synonymous with the message. When instructional designers believe that learning occurs through this communication process, they select an expositive instructional strategy. Most traditional educators favor this approach. It is a four-step process in which the instructor should (1) present information to (passive) learners, (2) test learners on their recall or understanding of the message, (3) present opportunities for learners to practice or apply the message, and (4) present opportunities for learners to generalize what they have learned to real situations or problems (Romiszowski 1981, 293). At the other anchor point on the continuum is a different philosophy about learning and instructing. To adherents of discovery learning, learning is intensely personal. Set in the intimate mental world of the learner, it results not so much from manipulation of environmental variables outside the learner as from the learner’s own internalized insight, reflection, and experience. When instructional designers believe that learning occurs through this experience-oriented process, they favor a discovery strategy for instruction. It is a four-step process in which the instructional designer will (1) structure opportunities for learners to receive important experiences and observe or reflect on them, (2) question the learners about the experiences and observe learner reactions, (3) help learners think about the general principles and significant emotional experiences they have experienced, and (4) structure opportunities for learners to apply what they have learned to actual situations and problems (Romiszowski 1981, p. 294). While expositive and discovery instructional strategies constitute more or less opposite anchor points on a continuum, there are many points in between. Instructional designers may select from numerous methods that are appropriate to achieve the desired outcomes of an instructional experience. Romiszowski’s summary of these methods is given next (p. 180). Strategy

Description

Impromptu discovery

Unplanned learning: no instruction was involved directly (for example, free use of a library or resource center).

Free exploratory discovery

Broad learning goals are fixed; otherwise, the learner may choose how to achieve the desired outcomes.

Guided discovery

Objectives are fixed; the learner is guided on methods, conclusion.

Designing Instructional Interventions

153

Strategy

Description

Adaptively programmed discovery

Guidance and feedback are given individually.

Intrinsically programmed discovery

Guidance and feedback are given according to preplanned program, based on the “typical” student.

Inductive exposition

The trainer “talks through” the discovery process.

Deductive exposition

Lectures.

Drill and practice

Rote reception learning: instruction demonstrates what to do and provides practice. No conceptual understanding must be involved.

Much attention has focused around another form of instructional strategy called action learning, which may also be an approach to carrying out the instructional design process itself. When action learning is used, learners are given a real-world problem, assembled in teams to solve it, and are permitted the freedom to experiment to find solutions. Instructional Strategy Based on Events of Instruction Another way to think of instructional strategy is based on the events of instruction and the conditions of learning, not on the philosophy of the instructional designer. In this sense, instructional strategy is rooted in assumptions about what does—or should—happen during any planned learning experience and about what type of learning the instruction should facilitate. Different instructional strategies must help learners acquire verbal information, establish cognitive strategies, develop intellectual skills, build motor skills, or appreciate new attitudes. To select instructional strategy instructional designers start by examining performance objectives to determine what type of learning is to be facilitated. They should choose instructional strategy based on the learning. If learners are to acquire verbal information, it will be necessary to discover a way to make that information meaningful to them. The instructional designer may adopt a strategy of fitting isolated information, like definitions, into some pattern, such as rhymes, mnemonics, or acronyms that will be meaningful to learners and will improve their retention. If learners are to be aided in establishing a cognitive strategy

154

Mastering the Instructional Design Process

or in changing their attitudes, they should be led through a process of discovery using the discovery strategies listed earlier. If learners are to develop intellectual or motor skills, expositive strategies are often appropriate. Once the overall instructional strategy has been selected, instructional designers should focus attention on each event of instruction. “The events of instruction,” note Gagné, Briggs, and Leslie (1979, 155) in a classic description, “are designed to make it possible for learners to proceed from ‘where they are’ to the achievement of the capability identified as the target objective.” Events of instruction constitute what should be done in a planned learning experience: instructional strategy, a closely related notion, constitutes how they will or should be done. To plan instructional strategy for a learning experience, instructional designers begin by identifying each step in a learning experience. Then, bearing in mind the learning being planned, they pose the simple question, how can that be done? Authorities on instructional design have devised many schemes for describing the events of instruction over a year. Pucel (1989) identified eight key instructional events based on his own independent research and the classic research of Ausubel (1962), Chase and Chi (1980), and Herbart (1898). To apply the results of their research, instructional designers should: 1. State the performance objectives for the learning experience to clarify: a. What is to be learned? b. How the learner can demonstrate the desired performance. c. How performance will be judged. 2. Explain the importance of the learning experience. 3. Provide crucial background information that the learner must have to achieve the performance objectives (“tell” the learner what to do and why). 4. Demonstrate the behavior (“show” the learner). 5. Guide practice (ask the learner to “do” it or apply it). 6. Allow for unguided practice (ask the learner to “do” it or apply it without benefit of extensive instructor feedback). 7. Evaluate the learner’s performance and knowledge base (“follow up” with the learner). 8. Provide feedback and direction for future learning.

Designing Instructional Interventions

155

Possibly more widely known than these eight steps are the nine key instructional events identified in a classic treatment by Gagné, Briggs, and Leslie (1979) and updated by Gagné, Wager, Golas, and Keller (2005): 1. 2. 3. 4. 5. 6. 7. 8. 9.

Capture the attention of the learner. Describe to learners what performance objectives are to be achieved. Help learners recall prerequisite learning. Present instruction to facilitate the learners’ achievement of the performance objectives. Guide the learners through the material so they meet the objectives. Prompt the performance desired from the instruction so learners meet the objectives. Give the learners feedback, and make suggestions for improvement so learners sense how well they are meeting the objectives. Evaluate how well learners are achieving the objectives. Work toward helping the learners retain what they have learned and apply it.

The appropriate instructional strategy for each event depends on the desired results. Hence, types of instruction aimed at helping learners acquire verbal information, establish cognitive strategy, or develop intellectual skills, motor skills, or new attitudes will require its own strategy within the planned learning experience. Those strategies are summarized in Table 9.2. Choosing Instructional Strategy and Tactics For instruction to be effective, instructional designers should be able to choose among many instructional strategies and tactics.

Choosing an Appropriate Instructional Strategy Although authorities on reception and discovery learning have usually been interpreted as favoring a single instructional strategy for every learning situation, there really is no one universally appropriate strategy. Any instructional strategy can achieve any performance objective. Likewise, any instructional strategy can carry out any instructional event. However, no one instructional strategy works uniformly well under all conditions.

156

Mastering the Instructional Design Process

TABLE 9.2. INSTRUCTIONAL EVENTS AND THE CONDITIONS OF LEARNING THEY IMPLY FOR THE FIVE TYPES OF LEARNED CAPABILITIES Capability Event 1. Capture the attention of the learner 2. Describe to learners what performance objectives are to be achieved 3. Help learners recall prerequisite learning 4. Present instruction to facilitate the learners’ achievement of performance objectives 5. Guide the learners through the material so they begin to meet the objectives 6. Prompt the performance 7. Give feedback to the learners 8. Evaluate how well the learners are beginning to achieve the objectives 9. Work toward helping the learners retain what they have learned and apply it

Intellectual Skill

Cognitive Strategy

Information

Introduce a change in stimulus

Describe the performance to be achieved and give an example Encourage learners to recall subordinate concepts and rules Give examples of concepts or rules to be learned

Inform learners of the kind of solution that is expected

Describe what question is to be answered

Encourage learners to recall related strategies and intellectual skills Give unique problems to be solved

Encourage learners to recall the context of the information Give the information in the form of propositions

Give cues to the learners

Hint at solutions

Link to a broader context

Have the learners apply the performance Affirm that the rule or concept has been applied correctly Learner demonstrates application of concept or rule

Request solutions

Have the learners provide information or other examples Affirm that information has been stated correctly Learner restates information in paraphrased form

Review the material periodically with learners, giving them various examples

Affirm that the solution to the problem is correct Learner originates a novel solution

Give the learners opportunities to grapple with different solutions

Link the material to other information

Designing Instructional Interventions

157

TABLE 9.2. INSTRUCTIONAL EVENTS AND THE CONDITIONS OF LEARNING THEY IMPLY FOR THE FIVE TYPES OF LEARNED CAPABILITIES, cont’d. Capability Event 1. 2.

3.

Capture the attention of the learner Describe to learners what performance objectives are to be achieved Help learners recall prerequisite learning

Attitude

Introduce a change in stimulus Give an example of what action is called for

Demonstrate the expected performance

Encourage learners to recall information and other relevant skills Give learners a choice in their actions

Help learners remember what to do

4.

Present instruction to facilitate the learners’ achievement of performance objectives

5.

Guide the learners through the material so they begin to meet the objectives

Give learners the opportunity to observe the model or choice of what to do

6.

Prompt the performance

7.

Give feedback to the learners

8.

Evaluate how well the learners are beginning to achieve the objectives Work toward helping the learners retain what they have learned and apply it

Have the learners describe what they would do in real or simulated situations Give the learners reinforcement based on their choice Learners choose the desired course of action as appropriate Give learners new opportunities to choose the desired course of action

9.

Motor Skill

Give learners what they need to perform—such as appropriate tools, equipment, or other resources Give learners the chance to practice and the chance to receive feedback about their performance Have the learners demonstrate performance Give learners feedback on what they chose Learners are capable of demonstrating the skill/performance Encourage learners to practice

Source: Adapted from the classic work of R. Gagné, L. Briggs, and W. Wager, Principles of Instructional Design (4th ed.) (Fort Worth, TX: Harcourt Brace, 1992).

To choose the strategy, consider the learners, the desired learning outcomes, the learning and working environments, and constraints on the instructional design process. If learners are inexperienced, instruction based on an expositive strategy is usually the most efficient approach. Exposition leads learners

158

Mastering the Instructional Design Process

through a subject at a uniform rate, with the pace set more by the instructor than by the learner. Experienced adult learners will often rebel against an expositive strategy. They often prefer a process of discovery that uses their own experiences and allows them to become involved in, and committed to, learning. The desired learning outcomes should also influence choice of instructional strategy. Learners should not be asked to acquire verbal information precisely as they are led to develop a cognitive strategy, intellectual skills, motor skills, or new attitudes. Learning experiences are of different kinds, and a different instructional strategy is appropriate for each kind. The learning and working environments also influence the choice of instructional strategy. If the two environments are the same, as with on-the-job training, an expositive strategy is usually most efficient; however, if they differ, a discovery strategy usually works best. The closer the relationship between conditions in the learning and working environments, the greater the likelihood that learners can apply on the job what they learn during instruction. Finally, constraints on the instructional design process should also be considered during selection of strategy. Of primary consideration are time and control factors. A discovery strategy requires more delivery than an expositive strategy. Learners must be led to reach their own discoveries. That takes time, since individuals learn at different rates. However, greater control is possible with an expositive strategy in which the instructor transmits the same information to all learners. There may be differences in how that information is received and interpreted. But an expositive strategy usually leads to greater control over outcomes than a discovery strategy in which learners reach their own independent conclusions about their experiences. Choosing Appropriate Instructional Tactics Tactics are the ways instructional strategies are implemented. They are the detailed approaches and activities used by an instructional designer to accomplish a strategy. In this respect, they bear the same relationship to instructional strategy as daily operational tactics bear to corporate strategy. Just as a corporate strategy of growth is achieved through such tactics as increasing sales, decreasing expenses, increasing market share, or a combination of all these, so, too, can instructional strategies of reception or discovery learning be achieved through various methods (tactics).

Designing Instructional Interventions

159

The choice of instructional tactics has often been more art than science. Instructional designers should first identify the results they wish to achieve through instruction and then plot out how they will achieve those results. Choosing tactics has usually been left to instructional designers’ creativity and imagination, whether they are working as individuals or on a team.

Choosing Media and Delivery Methods The variety of media and delivery methods may present unique challenges to instructional designers. The choice should be made carefully and be based on the medium used. Choosing Media To plan to achieve performance objectives, instructional designers should also choose a medium, or media, after selecting an instructional strategy. The term medium just means the way an instructional message is communicated to the learner. Although the term media has not always been used consistently by instructional designers, examples are easy enough to identify: books, programmed texts, computers, slides or tapes, videotape, and film. A media selection model, sometimes called just a media model, is a decision-making aid. It should guide selection of instructional media according to their instructional and cost effectiveness. Many media selection models have been devised to help instructional designers, such as the classics by Reynolds and Anderson (1992). However “half a century of research on media has yielded little in the way of general guidelines for media selection. That is, we are not able to conclude that one medium or combination of media is more effective overall, or even that one medium works better for a particular type of learning or category of subject matter” (Gagné and Medsker 1996, 181). The Range of Media Instructional media range from simple to complex. This distinction can be understood in two ways. First, a medium that requires little advance preparation can be simple, while one requiring much preparation can be complex. Direct experience—possibly on the job—is simple because

160

Mastering the Instructional Design Process

it requires little preparation. Second, a medium that appeals to only one sense can be simple; a medium appealing to more than one sense can be complex. The fewer the senses to which instruction appeals, the less need there is to be concerned about the effects on each sense and about how media can appeal to the learners’ senses in combination. The classification scheme below is listed from complex to simple media. The simplest media are placed at the bottom of the media “cone”; more complex media are placed at the top. This scheme is based on a classic list by Kemp (1985). Media

Examples

Combinations of media

Interactive video Multi-image and sound computer-based training Multi-image/video Multi-image/audio Microfiche/audio Filmstrip/audio Slides/audio Print/video Print/audio Video Film Computer programs (displayed)

Projected motion pictures Projected still pictures Audio recordings Nonprojected materials

Slides Compact disc recordings Audio recordings Job aids Photographs Diagrams Charts Graphs Flip charts Chalkboards Print materials

Designing Instructional Interventions

161

Media

Examples

Tangible objects

Models Objects/devices/equipment Instructors/speakers

Of course, new social media add more choices to those listed above and may need to be considered separately because such a large number of choices are available. Instructional designers distinguish among instructional methods, presentation methods, and distribution methods. How do they decide just which medium is best to achieve performance objectives? Unfortunately, there is no one right answer to this question. Substantial research has been conducted over the years to determine which media are most appropriate for achieving desired instructional outcomes and supporting instructional strategy. But that research has not led to firm conclusions. Any medium can achieve any performance objective. But not all media should be used in precisely the same ways. Media selection decisions may improve as artificial intelligence and expert systems are applied to the instructional design process. However, the best approach to media selection is to make a primary media selection decision for an entire learning experience first, and then make secondary media selection decisions for each part of the experience. Do that by asking questions and noting the answers. When making a media selection decision, ask these questions first: What are the desired outcomes of instruction? Do they provide clues about what medium or media to choose? Suppose that instruction is being prepared so learners “will be able to troubleshoot problems with a diesel engine when given diagnostic instruments and a diesel engine.” Given that performance objective, what medium is appropriate? In this example, the performance objective itself specifies that the learner will be “given diagnostic instruments and a diesel engine.” That phrase suggests that learners will use tangible objects (as identified in the media classification scheme presented earlier). The objects may, however, range from real to simulated objects. The same principle applies to other performance objectives. Consult them first to determine whether they imply the medium (or media mix) to use during the planned learning experience. When making a media selection decision, ask this question second: What constraints on time, equipment, staff skills, and costs affect this planned learning experience?

162

Mastering the Instructional Design Process

When making a media selection decision, ask this question third: How will the instruction be delivered? While that question cannot be answered until a delivery mode has been chosen (see the next section), instructional designers should remember some media are more appropriate than others for particular audiences. PowerPoint slides (or their equivalents in other media, such as Prezi) are frequently used in group presentations but are not very effective by themselves for individualized instruction. When making a media selection decision, a fourth and final question to ask is this: How often will this planned learning experience be revised? Obviously, it makes little sense to invest large sums in a medium that will be difficult to revise. Yet that can happen to those who opt for video-based programs, when their organizations have no video production facilities. An expensive consultant is hired, the video is prepared and edited, and it is outdated by the time it is shown. Some media—print materials, slide shows, and overhead transparencies—are relatively easy to revise. Other media may not be that easy or inexpensive to revise.

Selecting Delivery Modes To plan performance objectives, instructional designers should also choose a delivery mode. A delivery mode means the choice made about the conditions under which instruction is to be offered. Not to be confused with media or instructional strategy, delivery mode is synonymous with the situation that confronts learners as they learn. The range of delivery modes is not great. There are only four basic choices, according to a classic discussion of this issue (Ellington 1985): 1. 2. 3. 4.

Mass instruction involving many learners. Group instruction involving fewer learners. Individualized instruction involving only one learner at a time. Direct experience involving real-time learning, such as informal on-the-job training.

Make a selection of delivery mode based on the performance objectives to be achieved. (See Figure 9.1.) If many people share the same instructional need, select mass instruction. It is appropriate, for instance, when everyone in the same organization should receive the same instruction. If only some people, such as employees in one work unit, require instruction, select group instruction. It is often appropriate for introducing new work

Designing Instructional Interventions

163

FIGURE 9.1. ALGORITHM FOR SELECTION OF INSTRUCTIONAL MODE START What performance objectives do you want to achieve?

What basic instructional strategy(ies) do you think would be most suitable for achieving these objectives with the people you will be working with, taking account of all relevant factors?

Mass instruction

Individualized instruction

Group learning

Direct experience

Source: Taken from W. Rothwell and H. Kazanas, Mastering the Instructional Design Process: A Systematic Approach (4th ed.) (San Francisco: Pfeiffer, 2008), 249.

methods or new technology. If only one person experiences an instructional need, select individualized instruction. If the need is minor—not enough of a “chunk” of information to warrant preparation of a planned learning experience—then rely on such direct experiential methods as supervisory coaching or on-the-job training to supply learners with what they need to perform competently. Once the delivery mode for the entire learning experience has been selected on the basis of terminal performance objectives, reconsider media selection for each enabling objective. Allowing for Constraints on Choice of Delivery Mode Instructional designers rarely enjoy complete freedom to choose whatever delivery mode they wish. They face the same constraints when choosing delivery mode that they face when choosing media. Limitations of time, equipment, staff skills, and costs can and often affect choice of delivery mode. In addition, managers—and sometimes the learners themselves—will make their preferences about delivery modes known. For

164

Mastering the Instructional Design Process

instance, middle managers in some organizational cultures dislike mass instruction delivered during working hours because it removes workers from the production process and results in too much lost production time. They may require that mass instruction be delivered after hours or on weekends reducing the chances that all workers can participate in it. Likewise, some learners prefer individualized instruction because they have a storehouse of unpleasant memories from their formal schooling about classroom learning situations. When choosing delivery mode consider constraints and management and worker preferences.

Appreciating the Learner’s Perspective: A Brief Overview of Cognitive Strategies Just as much attention should be devoted to appreciating the learner’s perspective as the instructional designer’s perspective. Savvy instructional designers will think about cognitive strategies for learners that will encourage their information processing. Cognitive strategies are the learned ways that enable learners to manage their own learning. While this book focuses primarily on what instructional designers do, cognitive strategies regulate how learning progresses. Individuals acquire cognitive strategies from their experience and schooling. While various taxonomies have been suggested to describe cognitive strategies, Gagné and Medsker (1996) skillfully described how to encourage the conditions of learning for training. One way to view cognitive strategy is through the lens of the open systems model (described in Chapter One), which includes input cognitive strategies, process cognitive strategies, output cognitive strategies, and feedback cognitive strategies.

Input Cognitive Strategies An input cognitive strategy depends on what learners pay attention to. Learners may be stimulated to pay attention by events external to them, by their own choice, or by a combination. An example of external stimulation might include job loss, which would create a significant emotional event for learners that would stimulate their learning on the job search. An example of internal stimulation might include remembrance of career goals, which could motivate individuals to seek new approaches to meeting those goals.

Designing Instructional Interventions

165

Process Cognitive Strategies A process cognitive strategy helps learners make sense of what they learn. Gagné and Medsker (1996, 75) list several: • Rehearsal: trying out something new. • Elaboration: associating something new with something previously learned. • Organization: imposing a structure on what is newly learned through such methods as outlining, categorizing, or diagramming. Output Cognitive Strategies An output cognitive strategy means that learners acquire new knowledge or skill by applying what they have learned and making meaning of their experiences. An example would ask learners to prepare instruction on something they would like to learn. The teaching (output) focuses the learners’ attention on organizing the new knowledge or skill to teach it to others. That is an output-oriented cognitive strategy. Individuals could use the same approach to make sense of what they want to learn. Feedback Cognitive Strategies A feedback cognitive strategy means that learners acquire new knowledge or skill by giving feedback to others. An example would ask learners to hear a speech and provide feedback to another person about that speech. Giving feedback focuses the learners’ attention on organizing the new knowledge or skill to provide feedback to others. That is a feedback-oriented cognitive strategy.

Using Appropriate Message, Visual, and Motivational Design Principles This section addresses message, visual, and motivational design principles. Message Design According to Kathy Dye (1997, 1): A message is a discrete unit of data transmitted through various sensory abilities: sight, sound, touch, taste, and smell. When data is

166

Mastering the Instructional Design Process

received accurately, it becomes information. The transaction is called communication. Messages are sent for a variety of purposes: to inform, to entertain, to persuade, to anger, to soothe. Message design is the systematic and purposeful process of deciding about communication. This involves decisions about both the content (“what”) and the delivery (“how”) of a message. Most of us make message design decisions either unconsciously or from habit: whether we phone or send a note, whether we sit with our legs crossed or not, whether we speak loudly or softly.

As Dye continues to explain, good message design centers on: • • • • • •

Grid-based design Perceptibility (physiological and psychological) Chunking and sequencing content Spatial relationships (size and placement) Secondary messages Color

Grid-based design means that the information is presented in a consistent format. A printed page is, for instance, a consistent design. Perceptibility is related to human perception, which has the elements of sensory perception and cognitive perception. People need to be able to see, hear, smell, or touch. Anything that makes that difficult, such as insufficient lighting, will block sensory perception. Cognitive perception relates to understanding. Anything that makes it difficult for people to understand will affect cognitive perception—such as changing the meanings of words, use of jargon without definitions, or other barriers to clear communication. Chunking means that people must be given information in ways that will make it easy to understand—such as positioning on a printed page. Sequencing means there must be an order to a message so people receive essential definitions first and then build what they need to learn based on those definitions. Spatial relationships indicate how messages are presented relative to each other. For instance, pictures should be presented in ways that easily support print, audio, or other media. Secondary messages relate to redundancy and clarity—such as a picture that supports a print explanation. Colors help message information stand out. But colors have different psychological meanings that may influence what people perceive and how they perceive them. Exhibit 9.1 summarizes important issues in message design.

Designing Instructional Interventions

167

Visual Design According to usability.gov, visual design focuses on aesthetics. It is sometimes related to graphic design. It is creating a balance among images, colors, fonts, and other elements. An effective visual design makes it appealing to the user by balancing these and other elements. Critical issues in visual design include (see www.usability.gov): • • • • • •

Lines Shapes Color palette Texture Typography Form

While graphic design is a field of its own, instructional designers working in smaller organizations may need to do their own layouts and become somewhat familiar with graphic design. An excellent resource for that purpose is discussed next. Motivational Design How can learners be motivated? That is an essential issue in motivational design. According to Keller (2006), “motivational design refers to the process of arranging resources and procedures to bring about changes in motivation. Motivational design can be applied to improving students’ motivation to learn” (p. 3). It draws on empirical theories of human motivation. Addressing reasons for learner motivation—or lack of it—go all the way back to the beginnings of instructional design. Workers rarely participate eagerly in learning experiences because they do not see how or why the learning is useful or benefits them. Adults, it is said, are problem-focused in their learning and must see why something is worth learning. People also attend learning events for different reasons—for sheer love of learning, to solve a work or personal problem, to socialize with others, or some combination of these reasons. When learning events are designed in ways that do not appeal to these motivations, then learners lose interest. That could be one reason online learning often has high dropout rates—that is, learners are given insufficient opportunities to interact, to be challenged, or to see why (in practical terms relevant to them) they are learning what they are given to learn.

168

Mastering the Instructional Design Process

EXHIBIT 9.1. SUMMARY OF BASIC CONCEPTS OF MESSAGE DESIGN Concept

Definition

“Exemplary” Errors

Grid-based design provides necessary psychological order.

• a consistent pattern of organizing the components of a message • major differences are perceived as important • minor differences are annoying

• computer-generated presentations with 5 or 6 different backgrounds • different covers for modules in the same course

Perceptibility is essential for communication.

• two components: physiological • one VCR for a group of 50 people and cognitive • “7th generation” handouts • physiological perception (sensory stimulus is received) • passive voice, negative • cognitive perception (meaning explanations, jargon is received) • determined by learner (receiver)

Chunk content into manageable pieces, then sequence them in a logical order.

• a chunk is a discrete bit of the message • the sequence is the order of presentation • “manageable” and “logical” are relative to the learner

• a 5-volume Policy and Procedure manual for new hires • printing slide presentation for handout

Spatial relationships are part of the message.

• relationship of size reflects relationship of meaning; “bigger is better” (principle of size) • relationship of space reflects relationship of meaning (principle of proximity)

• squeezing text to “make it fit” • several detailed slides for a relatively unimportant point

Secondary messages should support the primary message.

• secondary messages include metaphors, graphics, models, sounds, illustrations, etc. • must be appropriate to be useful

• “dingbat” decorations • mixed metaphors

Color is emotional and powerful; use it carefully.

• physiological component: how the eye perceives color (contrast, color blindness) • psychological component: general emotional reaction, emotional meaning of colors.

• full-color media icons for leader’s guide • pastel screens on zany, interactive computer “game”

Source: K. Dye, “Message Design: A Key to Effective Instructional Materials,” 1997, http://kathydye.com/ IDEAS/MessageDesign2.pdf.

Designing Instructional Interventions

169

While many books have been written about motivation—and their principles can apply to instructional design—perhaps best-known is John Keller’s ARCS Model of Motivational Design. According to Keller, four issues should be addressed in motivational design. First is getting the learner’s attention. That can be done by inspiring their perceptions through surprise or through appealing to a problem to be solved. Second is demonstrating relevance. Show them why they should care about the learning. Third is confidence. That means build learner self-esteem by providing quick wins or by providing positive feedback on their efforts. Fourth and finally is to build learner satisfaction. Inspire them. More details about this important and useful theory can be found in Keller’s (2010) classic book Motivational Design for Learning and Performance: The ARCS Model Approach. For more on message and visual design, see Bishop (2014) and Darr and Seel (2014).

Accommodating Social, Cultural, Political, and Other Individual Factors Influencing Learning Learning and instructional design do not occur in a vacuum. Factors external to the learning experience influence it. Among them are the social, cultural, political, and other individual contexts. In a global world, instructional designers must remain sensitive to the issues that may help or hinder learning. It is wise to avoid references to politics, religion, or humor in most instruction in today’s business settings. Consider, for instance, humor. What is funny to some people may prove offensive to others. Perhaps a story is appropriate here. Years ago, one of the authors attended a training session conducted by a licensed professional psychologist with a Ph.D. He opened his onsite session with a joke. One group found the joke funny and laughed. A second group became angry but remained silent, sitting sullenly through the remainder of the session. They voiced their true feelings later on in the written session evaluations, where they blasted the presenter. A third group got up and walked out, complaining about the joke and the speaker to the organizer of the event. The point to the story is that some words, phrases, and concepts can be deeply offensive to some groups. It is wise to avoid humor unless it has been checked in advance. It is also wise to ask reviewers of instructional materials to pinpoint any words or phrases that may be problematic. That may

170

Mastering the Instructional Design Process

have to be done in each culture where instruction may be used, because translations do not always carry the same meanings. A second wise practice is to use cultural informants when working cross-culturally. Use more than one informant because some individuals are not sensitive to cultural issues in their own cultures. Several informants are more likely to pick up what one person might miss. Consider these 13 general tips when designing or delivering training in other cultures (Rothwell 2012, 28–42): 1. Be sensitive to other cultures but realize that human beings are more alike than they are different. 2. Do your homework before entering a new culture. 3. Work with reliable local partners and informants. 4. Pay attention to marketing. 5. Consider language and cultural issues. 6. Recognize differences in how people participate and how they expect to participate. 7. Be prepared to negotiate. 8. Realize that people in other cultures don’t always communicate as openly as in the United States. 9. Recognize the importance of personal relationships. 10. Be tenacious in your efforts. 11. When traveling, make personal safety the first priority. 12. Be cautious in what you take for granted with foreign training audiences. 13. Don’t take training technology for granted.

CHAPTER TEN

PLANNING NONINSTRUCTIONAL INTERVENTIONS

A

ccording to Instructional Design Competencies: The Standards (2013), instructional designers should be able to (Koszalka, Russ-Eft, and Reiser 2013, 50) “plan noninstructional interventions (advanced). This is an advanced competency and it includes three performance statements, all of which are advanced.” The performance standards include: “(1) Identify which, if any, noninstructional interventions are appropriate (for example, performance support, knowledge management, personnel selection, job redesign, incentive systems) (advanced); (2) Justify why noninstructional interventions are appropriate (advanced); (3) Create design specifications for noninstructional interventions (advanced)” (Koszalka, Russ-Eft, and Reiser 2013, 50). This chapter provides additional information about this competency. It provides useful information about identifying which noninstructional interventions are appropriate, justifying them, and creating design specifications for them. The world is full of many problems. But solving those problems can be reduced to two key options. One option is to change individuals by equipping them with new knowledge, skills, or attitudes. That approach calls for an instructional intervention. Another option is to change the organizational environment in which individuals carry out their work. Since management does (or should) control most environmental factors 171

172

Mastering the Instructional Design Process

of organizations in which people carry out their work and strive to achieve results, any solution that is not an instructional solution may be properly regarded as a management solution. Management solutions include everything that could be done to affect human performance in organizations other than offering workers instruction or training. Management solutions—which are sometimes called noninstructional or nonlearning solutions—address human performance problems through means other than training, education, or development. While most books on instructional design do not treat these solutions, stakeholders of the instructional design process such as senior executives and operating managers demand that instructional designers broaden their focus to provide performance consulting and avoid restricting themselves to instructional solutions alone. We include this chapter to give instructional designers a rudimentary knowledge of management solutions. Our goal is not to be comprehensive in our scope, since there are myriad ways that an organization’s decision-makers could address performance problems through means other than by offering instruction (see, for instance, the classic work by Langdon, Whiteside, and McKenna [1999], Rothwell [2015], and Rothwell, Benscoter, Park, Woocheol, and Zabellero [2014]). Nor can instructional designers expect to become specialists in all methods. Nobody can be everything to everyone. But instructional designers should know enough to diagnose root cause(s) and discover a list of possible solutions. From that point, competent assistance from other specialists may be warranted.

Identifying Appropriate Noninstructional Interventions Many management solutions have been identified to address the root causes of human performance problems. Once root cause(s) have been identified (Mager and Pipe 1999; Rothwell 2015; Stolovich and Keeps 1992), an appropriate solution should be selected based on stakeholder support, costs, and benefits. Although hundreds and perhaps thousands of solutions are possible, for this chapter we shall focus on five solutions only: (1) feedback methods, (2) job performance aids, (3) reward systems, (4) employee selection practices, and (5) organizational redesign. These five solutions were chosen because instructional designers have often used them—in isolation or in combination. However, 50 interventions have been well-documented (Langdon, Whiteside, and McKenna 1999). In our summary of each of the five key noninstructional solutions,

Planning Noninstructional Interventions

173

we will (1) describe what it is, (2) explain when it should be used, and (3) summarize how to apply the solution to human performance problems.

Feedback Options Feedback is a continuous process of providing information about an activity, sometimes during the activity itself. It serves two primary purposes. First, by stimulating people to continue doing more or less of what they are already doing, it influences the quantity of performance. Second, by stimulating people to change how or what they do, it influences the quality of performance. Feedback can be incidental, growing out of situations in a spontaneous way, or intentional, growing out of situations deliberately designed to provide people with evaluative information about how or what they do. Substantial research evidence exists to support the value of improving feedback. When Should Feedback Be Used to Address a Performance Problem? Use feedback after thoughtfully pondering these three questions (Rummler, 1983, 14): (1) Do employees receive enough information on the consequences of performing as desired? If the answer is no, provide feedback; (2) Are employees receiving accurate information on the consequences of performing as desired in a way that leads them to believe that their performance is correct? If the answer is no, improve the clarity and accuracy of feedback performers are receiving; and, (3) Are employees receiving timely information on the consequences of their performance so it can be used in time to improve what they are doing or how they are doing it? If the answer is no, improve the timeliness of the feedback. How Should Feedback Be Used in Solving Performance Problems? The quantity and quality of feedback can be addressed through coaching, production wall charts, memorandums, team meetings, performance appraisals, 360-degree feedback, and customer surveys. Each method, and others, can affect the quantity or quality of feedback that employees receive about what they do, how well they do it, what results they achieve, or how well their work results match up to desired results. Any approach that can improve the clarity and timeliness of feedback ranks as significant in solving human performance problems.

174

Mastering the Instructional Design Process

Coaching occurs during work activities and is appropriate for improving employee behavior on a short-term—even minute-by-minute— basis. Although employees can (and sometimes do) coach each other, coaching is often done by supervisors, who offer their employees timely, immediate, and concrete feedback about performance. Coaching sessions may last between a minute and many hours. Effective coaches are supportive, expressing through body language and spoken word their confidence that the employee is capable of superior performance. Effective coaches can also make a point quickly, reinforce the importance of the point, establish (perhaps on the spot) a plan for improvement with the employee, gain employee commitment and willingness to change, deal effectively with excuses, describe the consequences of human performance problems, and maintain confidence in employee abilities. That may include improving access to informal social networks. Executive coaching has emerged in recent years as an important performance improvement intervention to increase the interpersonal effectiveness of those in leadership positions. Production wall charts are visual displays that provide immediate, concrete feedback to employees about their performance, often daily. They are appropriate for increasing feedback on how much or how well individuals or work groups are producing. The typical wall chart in a manufacturing firm, as one example, might illustrate individual or work-group piece rates, error rates, scrap rates, and various other information on a graph. The typical wall chart in a service firm, as another example, might illustrate the results of customer perception surveys about the quality of customer service or the incidence of customer complaints. With the feedback provided by these charts, employees can see tangible results from their work and can change it based on the feedback they receive. Memorandums are short, written directives to employees. They provide practical, how-to-do-it guidance on handling common or unique problem situations. Often, they are prompted by a mistake made by an otherwise good, experienced performer and serve as feedback intended to change what employees do or how they do it. They may be delivered via print or electronic media such as e-mail. Team meetings are vehicles for giving feedback to all members of a work group about what they are doing or how well they are performing. Team meetings also provide a means of increasing group cohesiveness by building a sense of psychological closeness among members of a group. High group cohesiveness is equated with increased group performance when group goals coincide with organizational goals. Much attention has been

Planning Noninstructional Interventions

175

focused on methods of conducting successful team meetings—and “team building,” which is increasing work group cohesiveness. Performance appraisal is an excellent tool, in theory at least, for providing individuals with structured feedback. Just as job descriptions outline major job activities, appraisals measure how well employees carried out those activities. Performance appraisal is continuous—people are always appraising employee performance—even when no formal performance appraisal process exists in an organization. In many organizations, formal appraisals are often conducted on a cycle, usually once a year. While too infrequent to substitute for the spontaneous feedback provided by supervisors or coworkers to employees on daily work performance, appraisals are appropriate for uncovering and highlighting long-term performance trends and developmental opportunities. Unfortunately, performance appraisals are not always effective in providing structured feedback to employees. There are many reasons, and those reasons may provide clues to why nearly one-third of all HR practitioners were identified in one study to be unhappy with their organizations’ appraisal programs. Some appraisal systems try to accomplish too much. They may be intended to provide evaluative feedback on past performance, plan future career advancement, justify salary actions, and assess training needs. Sometimes the appraisal systems themselves lack top management support, fail to provide a method for establishing performance planning or standards at the beginning of the appraisal cycle, discourage give-and-take discussions between employee and supervisor in favor of one-sided meetings led by the supervisor, and lack clear job relatedness. To be effective, an appraisal system must overcome these common problems. One trend has been to move away from isolated performance appraisal to adopt an entire performance management system. While performance management is sometimes a term in search of meaning, it can mean a way to manage how the work is aligned with the organization’s goals, how the individual’s work is planned, how the work is monitored, and how work is reviewed. The implications of that approach are clear: to avoid performance problems by creating a system by which good performance is planned, tracked, and followed up. One special approach to gathering and giving feedback, used by many organizations, is 360-degree feedback. It takes its name from the number of degrees in a circle. Many approaches to 360-degree feedback rely on written or electronic instruments to collect perceptions about individuals from those surrounding them in a circle of acquaintances, including their supervisors, coworkers, subordinates, customers, and even family

176

Mastering the Instructional Design Process

members. However, the same idea can be applied by other means, such as by using group activities and even collaboration software. A key strength of 360-degree feedback is its face validity. Its results are often compelling evidence to those receiving them, and it can often motivate people to want to change. A key weakness of 360-degree feedback is that it is only as effective as the quality of information available to those recording their perceptions. If people do not know enough about the performer to offer good feedback, then their views may be flawed. Customer surveys provide feedback to all members of the organization— managers and employees—about how well the organization is meeting the needs of people it should serve. This information can be most useful in planning for future performance improvement, for both the individual and the organization. Customer surveys can be conducted by enclosing written questionnaires with products, telephoning customers some time after product (or service) delivery, visiting customers on-site, offering toll-free hot-line numbers for questions or help, or providing web-based surveys to an organization’s list of past customers. An entire range of strategies going well beyond mere customer surveys may obtain information about customer satisfaction, since customers rarely know the precise causes of suboptimal service they receive. Hunger for feedback has prompted many organizations to seek such customer information.

Job Performance Aids According to Joe Harless (1986), in a classic treatment, a job performance aid is “a mechanism that stores information external to the user, guides the performance of work, and meets these four requirements: (1) Can be accessed and used in real time (employed during actual performance of the task); (2) Provides signals to the performer when to perform the task or increments of the task (stimuli); (3) Provides sufficient direction on how to perform each task (responses); and (4) Reduces the quantity and/or time the information may be recalled (reduces access of memory)” (p. 108). In a memorable quotation that emphasizes the simplicity and value of job performance aids—sometimes called job aids—Harless (1985, 5) once remarked that “inside every fat course is a thin job aid crying to get out.” Job aids provide employees with guidance on how to perform in the work context. They cost less to prepare and use than training, and some experimentation has even been done in using job aids during instruction to

Planning Noninstructional Interventions

177

accelerate the learning process to reduce training time. Job aids are easier than training to revise under swiftly changing work conditions. Job aids can be used with training to help ensure transfer of training to the job. This is why canny instructional designers sometimes deliberately create “trainee workbooks” or “job checklists” that lend themselves easily to being taken out of training and used immediately on the job (Harless 1986). When Should Job Performance Aids Be Used to Address Performance Problems? Job aids are appropriate when the consequences of errors are great, procedures are complicated, work tasks are not frequently performed, the time for training is limited, and the budget for training is also limited. But they are inappropriate when employees have no time during work tasks to refer to them or when an employee’s credibility with customers will be undercut by referring to a job aid during performance of a work task. (Imagine a patient’s reaction if a medical doctor had to use a checklist to handle a medical procedure.) Nor are they appropriate when the consequences of errors are not great, work procedures are simple, and employees frequently perform the task. How Should Job Performance Aids Be Designed and Used? Virtually anything that provides on-the-spot, practical guidance can be a job aid, such as cues built into the questions on an application form that explain what information is being requested. Examples of job aids include cleaning instructions sewn into clothing, lights on automobile instrument panels, operators’ manuals provided with personal computers, and warnings on medicine bottles. However, the most familiar job aids include checklists, decision aids, algorithms, procedure manuals, and work samples. Recent attention has also focused around simple online job aids and around real-time performance support provided by instant messaging, social media, m(obile) learning, and other aids or approaches. Checklists are simple to design and are widely applicable to any activity (such as an organization’s procedures) that must be performed in a sequence. To create a checklist, begin by listing tasks of an activity or procedure in the order they are supposed to be performed. Label the column above the tasks “Tasks to Perform.” Then add another column for responses, such as “yes,” “no,” and “not applicable to this situation.” Label the column “Responses: Did You Do?” Be as short in your task descriptions

178

Mastering the Instructional Design Process

as possible to keep the checklist simple. If most employees are making the same mistakes, add notes to clarify tasks. Algorithms are usually visual representations, often resembling flowcharts, of steps to take in an activity or procedure. If employees follow an algorithm precisely, they should not be able to deviate easily from correct performance. Developing an algorithm closely resembles developing a checklist. Start with a task analysis and identify alternative actions in each step of an activity or procedure. Then flowchart the steps, depicting precisely what choices are available to a performer and what consequences will result from each choice. Use an algorithm only for short procedures, since lengthy ones will require many pages to flowchart. Procedure manuals are step-by-step instructions for carrying out work activities. They should serve as practical “how to” references, organized around typical work duties or problem situations, and can be consulted by performers as the need arises. To develop a procedure manual, begin with a comprehensive list of organizational policies or work-related problems. Conduct a separate task analysis or procedure analysis on each policy or problem activity. Then write step-by-step guidelines on what to do to comply with each policy or solve each work-related problem. Procedure manuals are often written with the aid of the play script technique, which takes its name from the highly structured scripts used in theatrical productions. A procedure described with the play script technique lists steps in chronological order from beginning to ending and uses columns with headings such as “When?,” “Who?,” and “Does What?” Items in the “When?” column describe the conditions under which action should be taken or the time it should be taken. Items in the “Who?” column affix responsibility for taking action. Items in the “Does What?” column describe what steps should be taken. While potentially useful for providing on-the-spot guidance to workers, procedure manuals are often tough to keep up-to-date. (When out of date, they create more of a performance problem than they solve.) When computerized they are usually easier to update. When placed online they become a computer-based referencing system. Work samples are examples of work that can be used by employees to save time or imitate a previously successful work product. It is easy to cite examples. Lawyers use work samples when consulting books or online sources filled with prewritten contracts. Secretaries use work samples when they search the web, download a letter, and revise it to handle a similar situation. Auditors use work samples when they prepare an “exemplary audit report” and then follow it as a template when subsequently asked

Planning Noninstructional Interventions

179

to prepare reports. If employees can see an example of something that has been done correctly, they can often replicate it closely in similar situations. That saves time, money, and effort while obtaining reliable results. Employee performance can be documented through work samples in online or paper work portfolios.

Reward Systems A reward system is the organization’s way of tying employee actions to positive consequences. You might think of it as the means by which an organization attracts people to join, keeps them working, and motivates them to train or perform. Rewards are the positive consequences that (presumably) greet individual performance consistent with organizational goals. Motivation means “the desire to perform.” It governs human choices of behavior and action (Vroom 1964), and it is related to rewards because people perform what they are rewarded for doing. While theories of rewards and human motivation differ, almost everyone agrees that employees do what they are rewarded for doing, avoid doing what they are punished for doing, and neglect what they are neither punished nor rewarded for doing. And managers are sometimes too limited in their thinking about how to reward performers for what they do. Managers do not always consider employee rewards or motivation when carrying out such typical management functions as planning, organizing, scheduling, delegating, controlling, budgeting, communicating, or staffing. Rewards do not always match up with desired performance, and the impact on performance is predictably negative. Organizations typified by a culture of “excellence” match rewards to organizational goals and desired results. Care must be taken to avoid unethical manipulation of human beings by using a mechanistic “carrot-and-stick” approach that promises rewards for performance and punishments for nonperformance. When Should Rewards Be Used in Addressing a Performance Problem? Rewards or work consequences should be reviewed when planning any change that will affect the organization, work group, individual, or job. Instructional designers should pose the following question before the change is implemented and consider the answer carefully: “What’s in it for the performer if he or she does what is asked?” To perform successfully,

180

Mastering the Instructional Design Process

people must feel they can succeed. They must also expect to receive a reward—and must value this reward (Vroom 1964). To complicate matters, however, individuals may vary in their perceptions of these issues. When troubleshooting existing human performance problems, instructional designers should pose these four questions to identify a problem caused by a poorly designed reward system (Mager and Pipe 1999): (1) Is the problem caused by obstacles in the work environment rather than by an employee’s lack of skills? (2) Before performing, does the employee expect not to be rewarded—or even to be treated negatively—because of performing as desired? (3) Are the consequences of performing with little perceived value to the employee? (4) Do employees find the consequences of performing as desired negative (punishing), neutral (no results), or positive (important)? If the answer to any (except for the last part of question 4) is yes, then the performance problem is attributable, in whole or part, to a poorly designed reward system. How Should Incentive Systems Be Used to Address a Performance Problem? In a classic article on incentive systems, Kemmerer and Thiagarajan (1989, 11) note that all incentive systems “should be intentional, external, and standardized.” By intentional, they mean that incentives should be deliberately designed to encourage a performance consistent with job or organizational goals. By external, they mean that rewards should be controlled and monitored by management levels within the organization. By standardized, they mean that all reward systems “should specify a standard procedure that identifies the employees, activities, and incentives—and the relationships among them” (p. 11). Employees in an organization may be categorized into groups or teams, the activities and accomplishments of each group or team may be identified as they contribute to organizational goals as measured by Key Performance Indicators (KPIs), and each group or team may be rewarded in line with its accomplishments. Whenever approaching a performance problem, instructional designers should always consider the consequences to performers of achieving results desired by an organization. Any intentionally designed incentive system may achieve from one to four possible goals: (1) contribute to attracting people to an organization, (2) encourage people to remain with the organization, (3) encourage people to behave in certain ways—such as follow standard operating procedures or apply creativity, or (4) encourage people to achieve work results desired by an organization.

Planning Noninstructional Interventions

181

If the consequences of performing result in none of these, then it is unlikely that the performance is being intentionally encouraged by the organization. But it should be, if the performance is valued.

Employee Selection Practices Effective employee selection practices involve matching people to work for which they are qualified. Employee recruitment, a related activity, involves seeking individuals qualified for the work and encouraging them to participate in the selection procedure. In these processes, managers in most organizations begin by analyzing work activities. They then infer from those activities the knowledge, skills, and attitudes for applicants to learn the job quickly, recruit people from sources where they can acquire the knowledge or skills, and screen individuals until the best-qualified candidate is matched to the work. Human resource managers are well equipped to provide insight into methods of improving selection and recruitment practices. If there is a single step that most organizations can take to improve human performance, improving selection and recruitment methods might well be it. Employee selection is an excellent leverage point to avoid human performance problems and/or to build on the organization’s competitive strengths. Instructional designers can help to do that by describing the work and/or the most successful people who perform it, analyzing that work and/or the most successful performers, and establishing selection criteria appropriate for choosing people best able to do the work. Selection methods influence training because the knowledge, skills, and attitudes that individuals bring to work influence what they must learn to perform competently. If experienced people are hired, training time should be reduced. The organization must pay a salary premium for experienced people. If inexperienced people are hired, training time should be increased. The organization can also pay less for salaries. When Should Selection Practices Be Used to Address a Performance Problem? Corrective action should be taken to improve organizational selection practices when most or all of the following symptoms are evident: (1) turnover is high; (2) involuntary termination rates are increasing from their historical rates in the organization; (3) employees complain that, when they were

182

Mastering the Instructional Design Process

recruited for, or placed in, their current positions, they were not expecting the work activities they subsequently encountered; and (4) supervisors and managers complain that their employees are ill-equipped, even after training, to perform duties for which they are accountable. How Should Selection Methods Be Used to Address a Performance Problem? If instructional designers have reason to believe that human performance problems in an organization stem in whole or in part from selection methods—or believe changes in selection methods can contribute to solving existing human performance problems or building the organization’s strengths—then they should focus their attention on each major step in the organization’s selection process. They should begin by examining recruitment, job or work analysis, selection tools, and selection results. Recruitment is attracting people to the organization. There are two labor pools from which to recruit: (1) inside the organization, and (2) outside the organization. Examine methods being used to recruit from both sources. Is any long-term, continuous effort made to target sources of talent, both internal and external, for entry-level vacancies, or do decision makers wait until vacancies exist and then scurry around madly looking for warm bodies to fill them? If the latter is the case, work to improve external recruitment by establishing internship programs with local schools, work-study programs with government agencies, and adopt-a-school efforts to build ties with local sources of talent. Run employment advertisements even when no vacancies exist to keep a large and current inventory of applications on file to use as the need arises. Make sure that recruitment efforts are targeted, as much as possible, at sources of talent appropriate for meeting the organization’s needs. Establish an employment brand by exploring why people remain with an organization rather than focusing solely on exit interviews as an indicator of why people leave. Establish internal job posting and career improvement programs so employees can gradually qualify for advancement in the organization. Recruitment may also be focused on two kinds of employment opportunities: long-term and short-term. Long-term workers are hired for extended time spans. Short-term workers are employed on a temporary or contingent basis. Examine methods used to select people for each kind of employment opportunity. Look closely at how short-term workers are matched up to work assignments and are briefed on what to do, how to do it, and why it is worth doing. If the work performance of short-term

Planning Noninstructional Interventions

183

workers differs dramatically from that of long-term workers, then try to improve screening and on-the-job training procedures. Job analysis is identifying work activities in the organization. The result of a job analysis is a job description, which literally describes what people should do. When addressing a performance problem that may be caused by poor selection methods, examine the completeness, accuracy, and currency of existing job descriptions. Do they provide clues, as they should, for identifying the knowledge, skills, and attitudes needed for successful job performance? Do they provide criteria for evaluating the education and experience of applicants relative to the knowledge, skills, attitudes and other characteristics needed for successful job performance? If not, work toward updating job descriptions or making them more complete or accurate. Start by consulting the Dictionary of Occupational Titles (see the online version at www.occupationalinfo.org/) or O’NET (at www.onet .gov). Then refer to other print and online references that can be helpful in preparing job descriptions. Although job analysis has traditionally (and unfortunately) been associated with rigidly defined and bureaucratically administered notions of “jobs” as “finite boxes full of work activities,” it can still be most useful in team-based organizations or in other settings where traditional notions of jobs are being supplanted by more flexible views of work design. In the latter settings, job analysis can be directed to team rather than to individual responsibilities and outputs. Selection tools are methods for structuring information or evaluating applicants relative to work requirements. They include application blanks, selection tests, and structured guides for interviewing job applicants. Written pre-employment tests have been the subject of substantial litigation since the Civil Rights Act of 1964 because the results might discriminate against various protected labor groups. Even though recent court cases have made the legal status of pre-employment tests equivocal, there is research evidence to support the desirability of using multiple selection tools instead of relying solely on unstructured job interviews. Sometimes, major improvements can be made in selection practices by substituting highly structured for unstructured employment interviews through a job performance aid called an employment interview guide. Selection results are consequences of recruitment and selection methods. Typical results may include separation (firings or resignations), retention in the present position, or movement within the organization. Take care to examine separations, both voluntary and involuntary, before

184

Mastering the Instructional Design Process

and after any change in selection procedures. Try to predict, in advance, what effects will be created by a change in selection methods. Selection results may also include the proportion of protected groups within the organization compared to those in the general population from which the organization recruits and hires. Socially responsible organizations support voluntary diversity-enhancing efforts to employ members of protected labor groups such as women, minorities, and the disabled. Many organizations take affirmative action to recruit, hire, train, and promote people in protected labor categories. Organizations may also support diversity programs to build appreciation for differences among workers. Any changes in selection practices should be made only after considering what effects (if any) they will have on efforts to support social equity in human resources practices. Approaches to selection have been complicated by a tendency to use more part-time or contingent workers. Some observers believe that “jobs”—in the sense of finite clusters of work activities—are a thing of the past. In some organizations, self-directed teams undertake an entire work process, and each worker achieves all goals established for the team. While that produces enlarged jobs, it also complicates defining what people should learn and how their individual performance should be judged.

Organizational Redesign Organization design refers to establishing reporting relationships and command structure. It determines who has authority to make what decisions and who achieves what results (Galbraith 2014). Organizational redesign is changing “assigned goals, responsibilities, and reporting relationships within a given organization” (Rummler 1986, 212). Although typically connoting changes in the organization’s structure (reporting relationships), organizational redesign in a broader sense may include any change in the structure of an organization, division, department, work unit, team, or job. It may incorporate job or work redesign, which involves changing what a job entails or how it is carried out. Job or work redesign can be synonymous with work restructuring. The research leaves little doubt that organizational design affects organizational and individual performance. Less certain is how much performance is affected by the personal motives of those establishing organizational designs, and what unpredictable results can stem from changes made to those designs.

Planning Noninstructional Interventions

185

When Should Organizational Redesign Be Used to Address a Performance Problem? Consider organizational redesign as a possible solution to human performance problems when the following eight symptoms are evident: (1) confusion about job responsibilities; (2) vague or unclear job descriptions; (3) outdated organization charts; (4) unclear relationships between the organization’s stated strategic goals and its structure; (5) complaints from supervisors and managers about overseeing too many people or too many jobs; (6) pockets of “burned-out” employees doing boring work, too much work, or too little work; (7) inefficient workflow, resulting in inefficient steps, unnecessary complexity, or other wasteful uses of resources; or (8) inability by the organization to adapt swiftly to dynamic conditions in the external environment, such as new competitors or unusual requests from customers or suppliers. How Should Organizational Redesign Be Used to Address a Performance Problem? Rummler (1986), in a classic treatment, outlines steps for instructional designers to follow in organizational redesign that remain useful. First, he suggests determining where there is a need to redesign the structure of jobs in the organization or the collection of activities a job comprises. Redesign should be considered, he notes, only when the organization is experiencing a performance problem in responding to external demands or in using resources efficiently. Second, he recommends examining the primary responsibilities of each major structural component of the organization—division, department, or work unit—to identify key problems affecting each component and describe the workflow passing through the organizational system. Third, he suggests preparing alternate and improved workflow models. Fourth, he emphasizes the importance of establishing a mission (purpose statement) and goals (desired results) for the new, major structural components illustrated on the organization chart. Fifth, he recommends drawing up a new organization chart (structure) for the organization based on environmental demands and efficient workflow. Sixth and finally, he urges that establishing new missions and goals down the organization’s chain of command be continued until each division, department, and job is included. Rummler’s suggestions for organizational redesign are logical. Similar suggestions are offered in the literature of strategic business planning,

186

Mastering the Instructional Design Process

business process reengineering, and job redesign. Unfortunately, organizational redesign is as much a political issue affecting the power of individual managers as it is an efficiency issue affecting an organization’s ability to survive in its environment. Logical approaches do not always prevail and are sometimes sacrificed to the whims of self-interested managers whose political goals may center on adding to their staffs, budgets, and prestige. Jacobs (1988), in a classic treatment, describes six ways to carry out organizational redesign. He suggests (1) changing reporting relationships; (2) improving information sharing; (3) defining job responsibilities; (4) changing job responsibilities; (5) changing goals, objectives, or standards; and (6) increasing information available about workflow systems. Changing reporting relationships, or reorganization, means altering who reports to whom. It is the one method most commonly associated with organizational redesign. It should be used carefully because changing the leaders of various organizational components can have unintended and negative side effects. For instance, subordinating one activity or department or manager to another inevitably reduces the emphasis placed on the subordinated activity or department and can create another management layer through which approvals must pass. Improving information sharing means finding ways to increase relevant, job-related information about workflow in an organization. To achieve this goal, conduct a communication audit. Using a standardized questionnaire and approach to analyzing organizational communication, examine what information—and how much—flows between work units. In addition, examine how and when information is communicated. Conducting such examinations can focus on e-mail (which is relatively easy to analyze) compared to interpersonal communication (which is not so easy). Defining job responsibilities must do with analyzing what people do, how they do it, and what results are desirable in line with organizational goals. When responsibilities or work goals are vague, employee and organizational performance can be improved by describing what people do. In practical terms, it means creating job descriptions when they do not exist, revising those that are outdated, or communicating to employees for what job responsibilities they are accountable. Changing job responsibilities is sometimes appropriate to address human performance problems caused by boring jobs. Use job enrichment to add tasks to jobs so they will become more interesting and will require employees to exercise increased responsibility. Job enrichment is a method

Planning Noninstructional Interventions

187

of creating a qualitative change in responsibilities. To address human performance problems caused by jobs with a limited range of tasks, use job enlargement to add more of the same tasks to the job. This is a means of creating quantitative change in responsibilities. To address human performance problems caused by shortages of staff in key positions—a common problem as organizations cut staff—use job rotation to relieve monotony and cross-train several workers for key jobs. Changing goals, objectives, or standards is a means of shifting accountability for a job, work unit, department, division, or organization. A goal is derived from a statement of purpose that addresses the reason for a job or organization. It is usually expressed in general, rather than in specific and measurable, terms. An objective is derived from a goal. It is specific and measurable. It describes what must be achieved in a period of time and how good achievement is defined. A standard is a minimum expectation of performance, usually expressed in measurable terms. By changing goals, objectives, or standards, decision makers can also change the direction of an organization or organizational component. Increasing information available about workflow systems means helping people understand how each part of an organization contributes to the products made or services delivered. There are many ways to achieve this purpose. In some organizations, for instance, expert systems guide nontechnical workers through the steps of answering a customer’s technical question or through troubleshooting common problems with equipment. In still other organizations, company websites run articles periodically on each department so employees will know what each department does and how work flows through it.

Justifying Appropriate Noninstructional Interventions When noninstructional interventions are judged by instructional designers to be warranted, they may have to be justified to sponsors and/or funding sources. Instructional designers should be prepared to mount a convincing case for their use and present that case to sponsors, funding sources, or other important stakeholders through a means likely to build support and understanding. One simple way to do that, while not the only way, is to brief stakeholders on when various instructional and noninstructional design choices should be selected. One popular approach is to use Thomas Gilbert’s performance engineering model, which isolates six categories of root causes,

188

Mastering the Instructional Design Process

as an organizing principle for a briefing. Then stakeholders are told what kind of interventions may be appropriate to address each category of root cause.

Creating Design Specifications for Noninstructional Interventions Just as instructional design interventions require design specifications that lay out project requirements, so also do noninstructional design interventions require specifications. A specification is nothing more than a detailed project plan. Like any plan, it should identify the problem, the solution, the measurable goals to be achieved, the action steps designed to achieve those goals, the timeline (with milestones) for implementation, evaluation methods and criteria, a list of staff members, and table of responsibilities of “who should do what—and when they should that.” Such specifications may be incorporated into project plans but should go beyond them to provide additional details such as measurable goals (which will make subsequent evaluation easier), evaluation methods (based on the goals), and a table of responsibility.

CHAPTER ELEVEN

SELECTING OR MODIFYING EXISTING INSTRUCTIONAL MATERIALS

A

ccording to Instructional Design Competencies: The Standards (2013), instructional designers should be able to (Koszalka, Russ-Eft, and Reiser 2013, 51): “Select or modify existing instructional materials (essential). This is an essential competency and it includes four performance statements, two are essential and two are advanced.” The performance standards for this competency include: “(a) Identify and select materials that support the content analyses, proposed technologies, delivery methods, and instructional strategies (essential); (b) Conduct cost-benefit analyses to decide whether to use or modify existing materials (advanced); (c) Validate selection or modification of existing instruction (advanced); (d) Integrate existing instructional materials into the design (essential)” (Koszalka, Russ-Eft, and Reiser 2013, 51). This chapter provides additional information about this competency. It offers advice on how to identify and select materials, conduct cost-benefit analysis, and validate the selection or modification of existing instruction.

Selecting, Developing, or Modifying Instructional Materials When instructional designers select, develop, or modify instructional materials, they: 1. Prepare a working outline. 2. Conduct research. 189

190

Mastering the Instructional Design Process

3. Examine existing instructional materials. 4. Arrange or modify existing materials. These four general steps are followed whether selecting, modifying, or developing instruction. Chapter Twelve will add to these steps, examining two subsequent steps that follow those listed above. Step 1: Prepare a Working Outline Before examining materials supplied by others, prepare a general working outline to achieve the objectives established for the learning experience. The objectives are the results to be achieved; the outline operationalizes those objectives, showing how they will be achieved. If materials are to be selected or modified from others, it is critically important to ensure that the outline covers what is unique to meet the organization’s unique needs. The outline need not be detailed, but it does need to emphasize what is most important. Adults can be uniquely tied to (some might say “fixated on”) their contexts. For instance, preparing a training program can be the task confronting instructional designers. Material drawn from a retail setting may have to modified heavily if it is to work in a manufacturing organization because learners will see only the contextual differences and not the conceptual similarities. Such an issue must be covered in the outline. Simply consider questions such as these in formulating the outline: • • • •

Who are the learners? Where are the learners functioning? (That is, geographical location) When are the learners functioning? (Is timing an issue?) How are learners functioning? (Is that important to meet the learning objectives?) • How important is it that some issues are covered? • How well does the outline address all issues covered in the objectives? • How is success in the program evaluated? Step 2: Conduct Research Conducting research, the second step in selecting or modifying instructional materials, is carried out to identify materials available inside or outside an organization. The cost of developing tailor-made materials is usually formidable. Instructional designers should not waste precious

Selecting or Modifying Existing Instructional Materials

191

time, staff, and money preparing these materials if they can be obtained from other sources inside or outside the organization. Begin research for instructional materials inside an organization by identifying knowledgeable people such as experienced workers, supervisors, union officials, top managers, human resource managers, or trainers. Ask them if they know of any unit, department, or division that might have had a past need (perhaps for on-the-job training purposes) for instructional materials like those necessary to meet the performance objectives. Describing the materials carefully, ask the knowledgeable people three questions: 1. Have you ever seen any procedure manuals, checklists, descriptive booklets, or training manuals on [subject name]? By asking this question, instructional designers may uncover instructional materials already available within the organization. These may be suitable for immediate use or may lend themselves to modification. Procedure manuals and checklists are the most common materials used to support on-the-job training (Rothwell and Kazanas 2004). They provide a solid foundation on which to base instruction, particularly if they are current and have already been field-tested. They have built-in credibility with operating management, especially if they have already been used successfully. 2. Who do you know in this organization who is especially knowledgeable about this subject? Use this question to link up with in-house experts who may know of instructional materials already available inside the organization. Focus attention on employees who perform training at the operating level. 3. What department(s) might have needed, in the past, to do special training on the subject? This question may help pinpoint the best places to look for materials in the organization. In some organizations, operating managers have authority to purchase their own training materials. Hence, useful instructional materials may be squirreled away somewhere, perhaps gathering dust, and they could be used, updated, or modified to meet other needs in the organization. Even when instructional materials are not available inside an organization, instructional designers may still transform existing work-related materials into instructional materials with minimal effort. Begin research for materials outside the organization in a different way. First, network with instructional designers in other organizations. Find them by using social media or professional associations. Contact colleagues through these organizations to determine whether they have previously had occasion to design or select similar instructional

192

Mastering the Instructional Design Process

materials. Ask to see their materials. Remember, however, that some organizations consider their instructional materials proprietary or copyrighted, so comply with any organizational protocols. If it is not possible to examine materials, then at least ask how they were prepared, focusing on the processes used rather than the products developed. Most instructional designers will share that information, even if they cannot share actual work products. (The favor may have to be reciprocated someday.) Searching print and computer-based references is another way to find existing instructional materials outside an organization. While few references will lead directly to instructional materials, many books or websites will be useful in sourcing them. Sometimes, instructional designers may even be lucky enough to find off-the-shelf instructional materials. Numerous references can be invaluable in conducting such searches. For instance, look for videos on the following websites or similar ones: • • • •

http://employeeuniversity.com www.videotraining.com.au www.trainingabc.com www.udemy.com

Numerous training materials can also be found for free, or for purchase, through searches on the web. (But note that copyright issues may sometimes be cloudy, so check with the website owner on fair use of all materials.) Step 3: Examine Existing Instructional Materials Evaluating existing instructional materials is the third step in selecting or modifying instructional materials. When debating whether to use existing instructional materials, be sure they follow the instructional strategy and performance objectives established for the planned learning experience. (Do not expect to plug-and-play existing materials to meet specific performance objectives without making at least minor modifications, even though relying on reusable learning objects may be possible.) Sometimes it is helpful to compare existing materials to a list of criteria on an evaluation checklist prepared in advance. When using such a checklist, think about these questions: (1) Can the existing instructional materials be used as they are, with minimal revisions? (2) What revisions must be made? and (3) Are the performance objectives to be met by learners so unique as to prevent use of anything except tailor-made materials?

Selecting or Modifying Existing Instructional Materials

193

Step 4: Arrange or Modify Existing Materials Arranging or modifying existing materials is the fourth step in selecting or modifying instructional materials. When existing instructional materials are appropriate to use, it may be necessary to secure copyright permissions and arrange or modify the materials in ways appropriate for satisfying the objectives. Securing Copyright Permissions. Copyright permissions must be secured for existing instructional materials whenever a copyright notice appears on the title page, or on a footer, of instructional materials. It is unethical for an instructional designer to do otherwise. (For an example of a copyright notice, see the page behind this book’s title page.) As a rule of thumb, assume that any material purchased from a vendor or borrowed from another organization is copyrighted. Most government documents are not copyrighted, though that must always be checked. To request permission to use the material, write directly to the copyright holder. State where the material will be used (in-house only?), how much will be used (the entire document or only part of it?), and how it will be used (in-house training only?). Will the material be adapted in any way? Also indicate how soon it will be used (is there an urgent need for a response?), how many copies are to be made (will all participants in training be given a copy?), who will receive the copies (who are the learners?), why the material is needed (training only? promotional use?), and how often future requests will be made (how many times will the material be used each year?). Be prepared to pay a (possibly hefty licensing) fee for the privilege of using the material. Arranging Instructional Materials. Even when existing instructional materials can be found to meet the needs of targeted learners, take care to arrange the materials suitable for the intended use. Modify the material, even if only cosmetically, so (1) it appears tailor-made to the industry and organization in which it will be used, and (2) it matches up exactly to the performance objectives established for the learning experience. Begin by making the most obvious changes. Revise titles and case study settings so they match up to the organization and learners. (Be sure that changes to be made to copyrighted material are noted in the permission request.) Then make major changes, if they must be made. It is often helpful to record each change on a point sheet. The point sheet, a lined document resembling a page of footnotes, is based on a syllabus and becomes a guiding document to help a team of instructional designers

194

Mastering the Instructional Design Process

tackle revision. Finally, arrange the materials so their format matches up to any special requirements favored by the organization in which they will be used. It can be useful doing this in front of a computer with two screens rather than one only.

Conducting Cost-Benefit Analyses to Decide Whether to Use or Modify Existing Materials Cost-benefit analysis is not one thing; rather, it is many (Phillips 2011). There are many ways to conduct it. The basic principle of cost-benefit analysis is simple enough. Compare the costs of something to the benefits of using it. If the benefits outweigh the costs, then it is worth doing; if the costs outweigh the benefits, then it is not worth doing. This approach is sometimes associated with so-called return on investments (ROI) to determine whether training is worth delivering. If the benefits (that is, the financial value of solving a problem) outweigh the costs of training (computed by totaling all costs associated with analyzing the issue, designing and developing instruction, implementing the instruction, and evaluating it), then training is worth doing. ROI may be forecasted before training is designed or after it has been conducted. The same principle may apply to deciding whether to “make” instruction (that is, design it) or “buy” instruction (that is, purchase from an external vendor such as an online provider, a consulting firm, a community college or university, or some other group). Much instructional material is readily available—some of it on the web—but rarely is so-called “off-the-shelf” instructional material so well prepared that it can meet all instructional needs for all audiences. The real question is this: will the benefits of buying and modifying instruction from an external source outweigh the costs associated with tailoring them to meet the unique needs for a targeted group? Several challenges exist when conducting cost-benefit analysis. The first is that it is much easier to pinpoint financial costs than it is to pinpoint benefits. Much has been published on determining costs, which can be relatively easy for a cost accountant. But less has been published on determining benefits—some of which may be nonfinancial (such as improved customer service, improved operational efficiency, or increased worker engagement). The second issue is that financial issues alone may not be sufficient to make decisions. Are there sufficient instructional designers available

Selecting or Modifying Existing Instructional Materials

195

to meet the organization’s demands? If not, high-priced instructional materials may be worthwhile even if they fail a financial cost-benefit test because trained staff are unavailable to meet stakeholder demands. Pleasing the internal customer may outweigh financial decisions.

Validating Selection or Modification of Existing Instruction Instructional designers often believe that instruction is not finished until the targeted learners can learn from the material. Concerned with helping formulate instruction, this step in the instructional design process calls for formative evaluation. Usually distinguished from summative evaluation, which helps summarize results of instruction, formative evaluation is conducted before instructional materials are delivered to a majority of the targeted learners. Summative evaluation is conducted after instructional materials have been used with targeted trainees and results have been measured. Evaluation in all its forms has figured prominently in recent treatments of instruction as decision-makers demand increasing accountability. A later chapter will examine formative evaluation in more detail.

CHAPTER TWELVE

DEVELOPING INSTRUCTIONAL MATERIALS

A

ccording to Instructional Design Competencies: The Standards (2013), instructional designers should be able to (Koszalka, Russ-Eft, and Reiser 2013, 53), “develop instructional materials (essential). This is an essential competency and it includes four performance statements, all of which are essential: (a) develop specifications that serve as the basis for media production (essential); (b) produce instructional materials in a variety of delivery formats (essential); (c) develop materials that align with the content analyses, proposed technologies, delivery methods, and instructional strategies (essential); (d) collaborate with production specialists (essential).” This chapter provides additional information about this competency. It offers advice on how to develop specifications as the basis for media production, produce instructional materials in various delivery formats, develop materials aligned with content analyses, proposed technologies, delivery methods, and instructional strategies, and collaborate with production specialists.

Developing Media Specifications Media specifications literally specify the media requirements for instruction. Think of them as a list of technical requirements to be met. They really 196

Developing Instructional Materials

197

resemble project plans for media use. While onsite instruction conducted in a classroom or individually may not require any media specifications, online or blended formats may require several. Not surprisingly, any media specification must answer several specific questions: • Who should use it? (Describe the targeted learners) • What is the justification for the use of media? (Why this media and not some other?) • How will the media relate to the performance objectives? • What kind of media should be used? (Print, web-based, video, audio, or some other) • When will they use it? (How is the media integrated with the remainder of the instruction?) • How will they use it? (What assumptions are made about what media the learner will need to access?) • How will the media relate to the performance objectives? • Where will learners use it? (What assumptions are made about the access point of learners? For instance, will it be available by touch-screen on the job?) • How long should it be? (Specify duration, which is particularly important for pricing video or some other media) • Why is the media justifiable? (Explain the reasons in terms of cost and learner effectiveness) • How will the media be used? (Explain application) • How much will the media cost? (Estimate the budget) While answering all these questions may not be necessary on all occasions, it is wise to consider them because savvy, cost-conscious clients and other instructional designers may question the use of expensive media when simpler, less expensive methods may be equally useful. Always consider the value of the media proposed against the costs and the likely learner benefits. Different media will have different requirements. These unique requirements should be reflected in the media specification document(s). A video may necessitate a script, a musical score, print documents, and perhaps even a storyboard that describes actors’ and actresses’ movements in the video. A video specification may also need to stipulate the format in which the video is to be presented (such as .avi, .mpg, or some other). The same principle applies to other media.

198

Mastering the Instructional Design Process

Producing Instructional Materials All previous steps in the instructional design process culminate in instructional materials that will help learners achieve desired performance objectives. Successful instructional materials share important attributes. In this chapter, we focus on designing instructional materials, offering advice to instructional designers about judging and justifying instructional materials they have prepared and identifying key ethical and cross-cultural issues in designing instructional materials. An Overview of Steps in Selecting or Designing Instructional Materials Instructional designers should regard selecting, modifying, or designing instructional materials as part of a larger process. Recall these steps from Chapter Eleven. This chapter focuses only on steps 5 and 6, as shown below: 1. 2. 3. 4. 5. 6.

Prepare a working outline. Conduct research. Examine existing instructional materials. Arrange or modify existing materials. Prepare tailor-made instructional materials. Select or prepare learning activities.

Prepare Tailor-Made Instructional Materials Preparing tailor-made instructional materials is the fifth step in designing instructional materials. This step should be carried out only when it is not possible to use or modify existing materials from inside or outside the organization. When approaching designing tailor-made instructional materials, think of developing a complete instructional package. Traditional Components of an Instructional Package. A traditional instructional package contains all the materials to tell learners what they need to know, show them what to do or how to use that information, allow them to practice what they have learned, and follow up with learners to give them feedback on how well they have learned. A traditional instructional package suitable for onsite (classroom) use has four distinct components according to a classic discussion of that topic by Dick and Carey (1990):

1. Learner directions or guide sheets are instructions for learners. They explain how to use the instructional package. In print-based media

Developing Instructional Materials

199

they sometimes take the form of a student manual and are important for individualized instruction. They are usually unnecessary when instruction is delivered in a group setting, since the instructor can provide learner directions orally. However, most instructors want to provide group participants with an organized manual, and that manual is a method of providing directions to learners. In electronically based media, the same purpose is served by instructions provided to learners. 2. Instructional materials contain the actual contents of instruction, including text and visual aids. They provide learners with the information they need to achieve the performance objectives. In electronically based media, they comprise lessons or all instructional “pages” or “frames.” 3. Tests are student evaluation tools. The term “test” is used in a broad sense. Types of tests include preinstructional assessments (pretests) to determine what learners know before they participate in planned learning experiences. Other types include self-check and instructor-check activities during instruction to determine how well learners are achieving enabling performance objectives (progress tests), and postinstructional assessments to determine how well learners have achieved the terminal performance objectives by the end of the planned learning experience (posttests). Further examples include job-based assessments (on-the-job performance tests) to determine how well learners are applying on the job what they learned in the instructional setting. Tests may be administered online, onsite, or in blended format. 4. Instructor directions or guide sheets are the instructors’ counterparts of learner directions or guide sheets. They are procedural guides to aid instructors in delivering instruction or in supporting learners as they individually apply themselves to planned learning tasks. Examples range from one-page instructor guide sheets or content and procedure outlines or lesson plans to voluminous trainers’ guides, trainers’ manuals, or tutors’ guides. Instructor directions or guide sheets are usually unnecessary in electronically based media, unless learners progress through instruction in groups, or instructors sit next to learners as they progress through computer-guided instruction. Differences of Opinion about Components of an Instructional Package.

Authorities on instructional design sometimes differ in their opinions about what should be included in an instructional package. One reason for this difference of opinion is that the authorities do not agree on one philosophy of learning and instructing (see, for instance, a description

200

Mastering the Instructional Design Process

of the theories in Reigeluth 1999a, 1999b, and 2008). The instructional designers’ underlying philosophies can have a major impact on how instruction is prepared, as numerous examples of widely diverse lesson plan formats show. Preparing Instructional Materials for Individualized Use. Usually, instructional designers should prepare materials for individualized use first and then modify them for group use. One reason to take this approach is that instruction designed for individualized use, with minimal instructor guidance, can save valuable instructor time. This time can then be devoted to those learners needing special help. A second reason: since group-paced instruction drags along at the pace of the slowest learner, it may keep most learners from excelling. A third reason to begin with learner-centered, individually paced instruction is that it usually requires more complete learner instructions than group-paced instruction, making modification for group use relatively simple. A fourth reason is that individualized instruction encourages learners to accept responsibility for their own instruction. This is not always true in group settings, in which learners too often play passive roles, while an instructor plays an active role as transmitter of information. A fifth reason is that online learning is often individually focused, and much instruction is moving to an online format.

How Should Instructional Materials Be Prepared and Formatted? Once instructional designers have decided what components of an instructional package should be used, they are ready to prepare and format the material. Preparing instructional materials is the process by which a working outline sketch is transformed into finished learner directions or guide sheets, instructional materials, tests, and instructor directions or guide sheets. This process is highly creative. Occasionally, it can often be made more efficient and effective by techniques such as detailed outlining or storyboarding. Detailed outlining is a step following preparation of a working outline or syllabus. A detailed outline summarizes the content of the planned learning experience—or series of related learning experiences—based on the instructional strategy and media selected earlier. Detailed outlining literally “adds details” to the simple working outline. Examples of such details might include visual aids, instructional material (handouts, text for the learner), and directions to the instructor or the learner. In this way, instructional materials are prepared directly from the working outline and

Developing Instructional Materials

201

are linked directly to performance objectives. At the end of this process the working outline has been converted into a detailed outline and, from that, into even more detailed learner directions or guide sheets, instructional materials, tests, and instructor directions or guide sheets. Storyboarding is a different method of preparing instructional materials. A storyboard is a visual representation, such as pictures of major frames in a videotape accompanied by the script text and musical score to go with them. However, storyboarding is not limited solely to visual media, although it is frequently associated with them. To create a storyboard, (a) find a large blank wall that can serve as the backdrop for the storyboard, (b) fasten a picture of each step in a designated instructional experience to the wall, and (c) develop accompanying learner directions and guide sheets, instructional materials, tests, and instructor directions or guide sheets for each picture. Each step in the instructional design experience must be represented visually; index cards or computer-generated overhead slides bearing text may serve this purpose, as well as pictures or murals. Instructional materials are then created to support each step. The value of the storyboard is that it helps organize instructional design efforts, whether they are performed individually or on a team. It also tracks what must be done in every step of the planned learning experience. Format means the print or audiovisual layout of instructional material in a medium. Choosing format means deciding about how the instructional message, and how instructor or learner directions and tests, will be organized and presented. It requires consideration of visual, message, and motivational design issues. Formatting learner directions, guide sheets, or student manuals is often of greatest concern when instruction is designed for individual use, since learners depend on this part of an instructional package to tell them what to do to progress through the learning experience. To prepare learner directions, some instructional designers like to start out by thinking of themselves as “novices” who are approaching the subject for the first time. This mindset is helpful because it forces them to view the material as the least experienced learner will see it. They then ask themselves this question: What step-by-step guidance do the learners and the instructors need to approach the planned learning experience through which the instructional package should guide them? Begin the learner directions or guide sheet with a few carefully chosen, one-phrase or one-sentence descriptions of the purpose of the learning experience and its performance objectives. Other topics that should be covered include the importance of the learning experience to the learner, its

202

Mastering the Instructional Design Process

relationship (if any) to other planned learning experiences or work tasks, prerequisites required, and necessary equipment or supplies. List, step by step, precisely what a learner must do to proceed through the instructional materials and tests. Once that is done, go back and list precisely what the instructor or tutor should do to prompt the learner through each step and place that information on an instructor (or tutor’s) guide sheet. When finished, the directions should be so clear that they could guide anybody, without a need to ask further questions about how to proceed, through the entire planned learning experience. Student manuals are formatted in two ways. One way is to set up the format so that the manual comprises many individualized learner guide sheets, each focusing on one lesson within related learning experiences. The manual begins with a program description, a statement of program purpose, terminal performance objectives, relationships among lessons (organization of the series), equipment and supplies needed to complete instruction, and self-check activities or tests to assess student progress. A second way to set up the format is to gear it toward participants in instructor-centered, group-paced instruction. In this format, the manual should also describe the program’s purpose, terminal performance objectives, and organization. In addition, it should contain highlights of the program contents, handouts, activities, tests, space for notes, and other material. Both formats can be used on or off the job. One advantage of giving participants a student manual in off-the-job instruction is that they can take it back to the job with them and use it as a job aid. (Some instructional designers prefer to use three-ring notebooks for the manuals to make revising material that much easier for them and adding or revising material on the job that much easier for learners.) Instructional materials have no one “right” format; rather, there are many possibilities. Examples may include lesson plans, audio or video scripts, and print-based or computer-based frames. The choice of format depends on the purpose of instruction, the performance objectives, who will use the instructional materials and why, how and where they are to be used, and what medium will be used for delivery of the instructional message. Instructor directions or guide sheets frequently take the form of lesson plans, detailed outlines intended to guide instructors through group or individualized instructional activities. A lesson is the most detailed level of instructional planning. Focusing on what instructors should do to facilitate a single planned learning experience, such as a class or tutorial session, lesson plans are essential to establishing the link between learners’

Developing Instructional Materials

203

achievement of desired performance objectives and instructors’ activities intended to foster that achievement. There is no one standardized format for a lesson plan. Many are acceptable. However, institutions may establish policies of their own on lesson plan format. A portion of a representative lesson plan appears in Exhibit 12.1. A lesson plan should usually be developed directly from an outline describing instructional content. It should reflect previous decisions made about instructional strategy, media, and sequence of performance objectives. It should be organized in several distinct parts, reflecting necessary instructional events for learning and the directions for an instructor to facilitate the planned learning experience. The instructional events become a guide for the parts of the lesson. Lesson plans should also specify the instructional resources, supplies, equipment, facilities,

EXHIBIT 12.1. A PORTION OF A REPRESENTATIVE LESSON PLAN (This lesson plan format is designed for group delivery.) Suggested Procedures

Notes for the Instructor

Display Visual 1.

Welcome the participants to the Workshop on Employee Incentive Programs. Describe the purpose of the workshop:

Purpose of the workshop To review methods of increasing employee involvement by tying rewards to work methods and results. Visual 1 Display Visual 2.

Describe the terminal performance objectives of the workshop:

Objectives of the workshop When you complete this workshop, you should be able to 1. Define the term Employee Incentive Program. Visual 2 Source: Taken from W. Rothwell and H. Kazanas, Mastering the Instructional Design Process: A Systematic Approach (4th ed.) (San Francisco: Pfeiffer, 2008), 272.

204

Mastering the Instructional Design Process

and other support materials needed for the planned learning experience. In this way, instructors, who may not be the same as the instructional designers who prepared the material, know what to do and how to do it. Scripts are similar to lesson plans in that they can establish the link between learners’ achievement of desired performance objectives and instructors’ activities intended to foster that achievement. They may be word-for-word texts of what an instructor will say to learners in a group, what a tutor will say to learners individually, or what will be said in electronically based presentations on videotapes, films, slide and tape shows, or audiocassettes. Like lesson plans, effective instructional scripts should be organized into distinct parts reflecting the events of instruction. If instruction is presented in an electronic medium such as video, the script should also provide directions for camera and background music. Formatting instructor directions or guide sheets or trainers’ guides is of greatest concern when instruction is designed for group use, since trainers depend on this part of an instructional package to tell them how to facilitate a planned learning experience. Any learning package requires directions to let instructors or tutors know what support they should provide to learners. For learning packages geared to individualized use, simplified instructor directions are usually adequate. These directions should describe the purpose of the package, the performance objectives, the structure of the learning experience, the resources, equipment, and facilities for the experience, and (most important) an overview of what learners must do to use the package. Instructor directions may be useful when they identify the most common difficulties encountered by learners during their individualized experiences, and tips to guide instructors to help overcome those difficulties. Trainers’ guides are usually necessary for group-oriented classroom instruction. They may take many forms and range from simple two-page brochures or outlines to three-ring notebooks filled with detailed lesson plans and everything else a trainer must deliver during a classroom presentation. Detailed trainers’ guides, such as those published for workshops, typically contain more than one lesson. One excellent approach to formatting trainers’ guides is to purchase and review several examples. Below, I summarize the parts of such a guide. Sections and Contents • Overview Performance objectives Module outline

Developing Instructional Materials

•

•

•

• •

205

Transparency/PowerPoint™ master list Handout master list Suggested training time Introduction Description of needs assessment Needs assessment questionnaire Training materials and aids Delivery preparation checklist Description of follow-up procedures Questionnaire for follow-up Related Materials A list of books, articles, videotapes, and other aids to be used in delivering the workshop Trainer’s Lesson Plan for the Workshop Lesson outline (points to be covered during the training session) Instructional notes (directions and information for the trainer) Slide Masters Master copies of slides Handout Masters Master copies of all handouts for the workshop

Research on trainers’ guides has shown that, under certain conditions at least, an outline format can be just as effective as a detailed format (McLinden, Cummings, and Bond 1990). A detailed format is one in which the instructor is given all information on what to present and the resources to make the presentation. An outline format is defined as one in which the instructor is “provided necessary content and sequence; however, scripted presentations, directions, and cues are kept to a minimum” (McLinden, Cummings, and Bond 1990, 3). The outline format saves substantial time and expense in materials preparation, but it presupposes that the content of instruction is nontechnical and that the instructors are prepared, highly experienced, and experts in their subjects (McLinden, Cummings, and Bond 1990). Tests should be formatted on the basis of the learner assessment methods chosen. While most novice instructional designers associate testing with paper-and-pencil or computerized assessment instruments, there are numerous ways to test knowledge, skills, and attitudes. Testing may occur through one-on-one questioning of learners, one-on-one demonstrations of ability during or after instruction, or questionnaires to assess changes in learner attitudes. Considerable creativity should be exercised when

206

Mastering the Instructional Design Process

formatting tests. One reason: the word “test” itself makes representatives of management, union, or workers very nervous, since they wonder how test results will be used in subsequent personnel decision making. Select or Prepare Learning Activities There are two general categories of learning activities: individual and group. How is each selected or prepared? This section addresses that simple but profound question. Individual Learning Activities. Individual learning activities are geared to individualized instruction and informal learning. Compared to the wealth of writings available on group learning activities, relatively little has been written about preparing individual learning activities on or off the job. There are, however, time-honored sources to which instructional designers may refer when they undertake preparing individual learning activities or helping people structure their own learning projects (Houle 1961; Knowles 1975; Tough 1979). Another approach is to provide learners with strategies for “pulling” instruction out of unwilling co-workers or supervisors too busy to provide it. Some learning management systems (LMSs) allow learners to choose from a broad range of learning activities to build competencies, measured by behavioral indicators, to populate their Individual Development Plans. Some learning activities are on-the-job; some are social or peer-based; and, some are planned learning experiences such as online or onsite learning (see Rothwell and Graber 2010). Almost any experience can be transformed into an individualized learning activity, provided that (1) outcomes are specified in advance, (2) the outcomes can be compared to pre-established performance objectives, and (3) the experience meets certain requirements from the standpoint of the learner or the instructor. It should furnish the learner with new information or skills, give the learner an opportunity to observe others applying a skill, allow a learner to demonstrate knowledge or skill, or afford an instructor a chance to assess how well the learner has acquired information or skills. Examples of individualized learning activities may include:

• • • •

Reading a book Interviewing others Reviewing documents Addressing a group on a new topic

Developing Instructional Materials

• • • • • • • •

207

Finding a problem Researching a subject Watching a videotape Observing others Demonstrating a skill Performing a new job Starting something new Solving a problem

Numerous others are possible. Even off the job, informal life experiences such as serving as a community volunteer, civic or church leader, parent, or spouse can become learning experiences and may serve job-related instructional purposes. One idea is to begin with a competency model and then, for each behavior or output listed to demonstrate the competency, provide a range of approaches to build the competency—such as watch an article, read a book, attend a workshop, visit a website, find a mentor who has an excellent reputation for that competency, take a field trip to a center of excellence well-known for showing the competency at its best, and so forth (Rothwell, Graber, Dubois, Zaballero, Haynes, Alkhalaf, and Sager 2015). To prepare individual learning activities, first decide how much instructor involvement will be necessary during the learning experience. If instructor involvement is necessary, supplement learner materials with instructor directions, guide sheets, or “tutor aids” so learners can be provided with instructor help as needed. For instance, instructor directions are important to support learners progressing through planned learning experiences in off-the-job, in-house learning centers. However, if instructor support is unnecessary, then use contract learning to guide individualized learning experiences. Contract learning is defined, according to one classic source, as “an alternative way of structuring a learning experience: it replaces a content plan with a process plan. Instead of specifying how a body of content will be transmitted (content plan), it specifies how a body of content will be acquired by the learner (process plan)” (Knowles 1986, 39–40). To be effective, according to Knowles (1986, 38), a learning contract should specify: • The knowledge, skills, attitudes, and values to be acquired by the learner (learning objectives). • How these objectives are to be accomplished (learning resources and strategies). • The target date for their accomplishment.

208

Mastering the Instructional Design Process

• What evidence will be presented to demonstrate that the objectives have been accomplished? • How this evidence will be judged or validated. Group Learning Activities. Group learning activities are perhaps most frequently associated with experiential instructional methods in classroom settings. While results of research studies on the relative effectiveness of group learning activities in classroom instruction have proved largely inconclusive, some group learning activities are better suited than others for meeting types of performance objectives. The choice of what learning activity to use should stem from the match between the performance objective and the activity. Numerous group learning activities can be identified. One enterprising author at one time catalogued and described over 350 (Huczynski 1983). In the classic book Approaches to Training and Development, Laird (1985) points out that some group learning activities are superior to others for giving learners the chance to become involved in, and committed to, the learning process. In the following paragraphs, we will provide brief descriptions of many common group learning activities, simple guidelines for developing them, notes about conditions when they are appropriate to use, and sources of additional information about them. A panel discussion is an assembly of knowledgeable people who meet with learners to deliver short presentations or answer questions about issues with which the panelists are familiar. A panel discussion is appropriate for helping learners with verbal information; it is inappropriate for providing instruction on cognitive strategies or for changing attitudes. To prepare a panel discussion, identify knowledgeable people who can speak on the issue and contribute toward achievement of the predefined performance objectives. Provide the panelists with a list of questions, or ask the participants to do so. Then identify an individual who can serve as panel leader to introduce panelists, pose questions to them, and keep the discussion on track. A case study is a narrative description of a situation in which learners are asked to identify or solve a problem. It is appropriate for instruction focused on cognitive strategy. Much has been written about it. To prepare a case study, first identify its purpose, the performance objectives it should support, and the targeted learners. Then conduct research inside and outside the organization. Try to find existing case studies that have already been prepared and field-tested by others.

Developing Instructional Materials

209

EXHIBIT 12.2. AN INTERVIEW GUIDE FOR COLLECTING CASE-STUDY INFORMATION Directions to the Instructional Designer: Use the questions appearing on this interview guide to help you collect information for a case study. First, find one or more experienced workers from the targeted job class for which instruction is to be designed. (Supervisors may also be used.) Then explain what kind of situation you are looking for—and why. When you find respondents who can think of example(s), ask the following questions. Finally, write up the case study and ask one or more respondents to review it for accuracy. Use disguised names, job titles, locations, and other facts. Add questions at the end of the case, if you wish. 1. What was the background of the situation? Where and when did it occur? Who was involved? Why was it important? 2. What happened? (“Tell me the story.”) 3. What caused the situation—or problem(s)—in the situation, so far as you know? (Describe the cause.) 4. What were the consequences of the situation for the people in it? the work unit? the department? the organization? (Describe the consequences.) 5. What conclusions can be drawn from the situation? What should be learned from it? If it happened again, how would you handle it? Why? Source: W. Rothwell and H. Kazanas, Mastering the Instructional Design Process: A Systematic Approach (4th ed.) (San Francisco: Pfeiffer, 2008), 278. Copyright © 2008 by John Wiley & Sons, Inc. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com

Look for those in books and articles on the web. If existing case studies cannot be located, then interview experienced workers in the organization to find examples of real situations demonstrating problems pertinent to the planned learning experience and supportive of its performance objectives. Use the interview guide appearing in Exhibit 12.2 to help structure questions that will produce the skeletal basis for a case study. As a last resort, prepare a fictitious case to serve the intended purpose. Use settings and characters compatible with the organization in which the instruction will be delivered. Then present the draft case to supervisors and workers for their review. Revise it according to their suggestions to make it realistic. Occasionally, this approach will help the reviewers remember actual situations suitable for case study treatment. Actual situations may then be substituted for the fictitious ones. (“Actual situations,” when labeled, have great credibility with learners. It is important, however, to conceal names to avoid embarrassing anyone.)

210

Mastering the Instructional Design Process

An action maze is “a printed description of an incident, for analysis, followed by a list of alternative actions” (Malasky 1984, 9.3). As learners suggest what actions to take, they are directed further in the action maze to find out the consequences of their decisions. This approach is effective for training people to troubleshoot problems and decide. To construct an action maze, begin as you would in preparing a case study. First, identify the purpose. Second, clarify how the action maze will help learners achieve performance objectives, providing them with new information or affording them an opportunity to try out their skills and receive feedback about what they know or do. Third, select a situation requiring decisions to reach a conclusion, such as a procedure comprising related tasks. Fourth, write up each step of the procedure to a decision point in which learners must choose what to do. Fifth, give participants two, three, or four choices only. Sixth, prepare a separate sheet describing what happened because of that decision and leading the learners to another decision point. Seventh, complete enough sheets to reflect the entire procedure, with or without “detours” made by novices. Eighth, request experienced workers to progress through the action maze to test how “realistic” it is. Ninth, revise the action maze based on the suggestions offered by experienced workers. An in-basket exercise “is a variation of a case study. Each participant is provided with an in-basket, including correspondence, reports, memos, and phone messages, some of which may be important to the case or process under study, and some of which may be extraneous” (Malasky 1984, 9.13). It is a timed exercise intended to discover how well each participant can manage details and withstand stress. Use an in-basket exercise only for “office skills” and supervisory practices. Do not use it for technical training focusing on heavy equipment or application of shop-floor procedures. To prepare an in-basket exercise, begin as preparation for a case study would begin. Identify the purpose. Then clarify how the exercise will help learners achieve performance objectives. Select or create memos, letters, or phone messages that require decision making and priority setting. Train experienced workers to observe participants and evaluate the quality of their decisions, providing feedback and coaching after the activity is completed. A role play is a dramatic representation of a real situation. It is an umbrella term for a whole range of similar group-oriented experiential activities. A role play gives learners an opportunity to prepare for situations they may face, or have faced, on the job. This group learning activity is useful for helping participants demonstrate and practice what they have

Developing Instructional Materials

211

learned during instruction. However, participants sometimes find role plays artificial and have trouble feeling and acting as they say they would on the job. Prepare a role play by writing a case study and then adding character descriptions to the case. Spell out exactly what learners should do during the role play. Use the framework for a role play in Exhibit 12.3 as the basis for preparing one. Fill in the blanks with information obtained from interviews with experienced workers. As in preparing case studies, base your role plays on actual situations confronted by workers. If that is not possible, then imagine realistic but fictitious situations. EXHIBIT 12.3. FRAMEWORK FOR EXHIBITING A ROLE PLAY Introduction

Use this role play to help you __________________________ ____________________________________________________.

Purpose of the role play Objectives of the role play Time required for the role play Number of people required for the role play Equipment/seating required for the role play Procedures Step 1: Step 2: Step 3: Step 4: Step 5: Step 6:

To do _________________________________. At the end of this role play, you should be able to: Spend ______________ (Please fill in with the time requirement) minutes on this role play. This role play is intended for groups of _______. To enact this role play properly, you will need a room with the following configuration and number of chairs: Assemble in groups of ______ for ______ minutes. Choose someone to play the part of each character. (Note how long that will take.) Read a description of the situation. Carry out the role play. (Indicate how long that will take.) Prepare for discussion. (Indicate how long that will take.) Ask participants to draw conclusions from what they learned and indicate how they will apply on their jobs what they have learned.

Mastering the Instructional Design Process: A Systematic Approach, Fourth Edition. Copyright © 2008 by John Wiley & Sons, Inc., p. 280. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com

212

Mastering the Instructional Design Process

A simulation is an artificial representation of real conditions. It may be computerized, or it may be prepared in print form. It should assess previous learning or demonstrate technical ability. Simulations are advantageous because they provide hands-on experience and are engaging to participants in a planned learning experience. But they have disadvantages. Among them: (1) they are usually expensive to develop, and (2) they may require an instructor to play the role of evaluator. Prepare a simulation by preparing a case study and then creating detailed descriptions of characters in the simulation. Spell out the purpose and objectives first. Then set the parameters of the simulation: how long it will last, where the simulation is to be conducted, who will do what, and when the simulation should end. Allow participants a measure of freedom so this extended role play feels realistic. Test out the simulation before using it to make sure it works. Revise it so it has reasonably predictable outcomes that are pertinent to achieving the performance objectives of the learning experience. The critical incident technique (CIT), sometimes called an incident process, is the production of a brief narrative description of a problem or situation. Often compared to the case study method, the CIT is appropriate for developing learners’ troubleshooting, decision-making, and questioning skills. It has been used in assessing needs and in delivering instruction. For more information about the CIT, see the classic description by Flanagan (1954). To prepare a critical incident description, interview experienced job incumbents performing the same work as the targeted trainees. Ask them to identify the most common or the most important (critical) problem situations (incidents) that they have heard about or experienced in the past. Then ask (1) how the situation was handled, (2) what results were obtained, (3) how the situation should be handled if it should come up again, and (4) what results should be obtained by using the recommended solution. From this information, create one- or two-sentence critical incidents based on real situations. Use the interview guide appearing in Exhibit 12.4 to help gather critical information. Another approach to preparing descriptions of critical incidents is to ask experienced job incumbents or their supervisors to keep performance logs to identify common or serious problem situations encountered during work. Use the performance log to identify how often problem situations are encountered by job incumbents and obtain detailed advice on how to handle them from exemplary job incumbents or their supervisors. Find out (a) the circumstances of the problem, (b) how the situations are handled,

Developing Instructional Materials

213

EXHIBIT 12.4. AN INTERVIEW GUIDE FOR GATHERING INFORMATION ON CRITICAL INCIDENTS Directions to the Instructional Designer: The questions appearing on this interview guide are intended to help you gather information about critical incidents for use in preparing experiential activities for instruction. First, find one or more experienced workers—or supervisors of those workers— from the targeted group of trainees. Second, ask them the questions that follow. Third, use the results of a series of these interviews to prepare critical incident activities for the targeted trainees. If you wish, prepare a second sheet showing the answers or recommended solutions for each critical incident. Ask the trainees to work on the incidents individually or in a small group. 1. Think back to a time when you faced a difficult problem on the job, perhaps the most difficult situation you had ever faced. (Describe, briefly, the nature of that situation.) 2. What did you do in that situation? What solution or approach did you use? (Describe it briefly.) 3. What happened as a result of your solution or approach? (Describe the results.) 4. Suppose this situation arose again. What would you do now? Why? (Describe a recommended solution and reasons for suggesting it.) 5. What results would you expect from using the solution or the approach you suggested in response to question 4? (Describe what you would expect the consequences of your action to be.) Mastering the Instructional Design Process: A Systematic Approach, Fourth Edition. Copyright © 2008 by John Wiley & Sons, Inc. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com

(c) what happens because of that solution, (d) how the situation should be handled if it should come up again, and (e) what results should be obtained by using the recommended solution.

Producing Instructional Materials in Various Delivery Formats Instructional designers who work in multimedia may find it helpful to think of an orchestra. Like a conductor who facilitates all the instruments to play in harmony, the instructional designer must blend media. Since it is rare for any one individual to be an expert in all media that may be used in instruction, preparing instructional materials in multimedia may require a team of such experts working together.

214

Mastering the Instructional Design Process

This basic principle can provide guidance when working to integrate print, video, audio, and even social media into a coherent whole. A storyboard format can be most helpful in doing that, just as it is used in professional filmmaking by Hollywood producers who must integrate oral script (what actors and actresses say) with music, camera shots, and special effects.

Developing Materials Aligned with Content Analysis, Proposed Technologies, Delivery Methods, and Instructional Strategies A special challenge is that, when working in multimedia, it is easy to lose sight of the overarching project goals and focus on the minutiae of each media and each instructional media. Maintain a schedule of regular meetings, onsite or online, to calibrate the work of all team members. Failure to do that will mean that some media are given preference over others or the larger project/instructional goals are lost in the shuffle.

Collaborating with Production Specialists It is rare for one team to do everything. A team of instructional designers in one nation might design an instructional program but others will deliver it. That means that teams of specialists in media may have to work together virtually to finish a program and perhaps others will roll it out. Building collaboration among production specialists is critical and requires excellent project planning and regular check-in meetings.

CHAPTER THIRTEEN

DESIGNING LEARNING ASSESSMENTS

A

ccording to Instructional Design Competencies: The Standards (2013), instructional designers should be able to (Koszalka, Russ-Eft, and Reiser 2013, 54) “design learning assessment (advanced). This is an advanced competency and it includes three performance statements, two are essential and one is advanced.” They include: “(a) identify the learning processes and outcomes to be measured (essential); (b) construct reliable and valid methods of assessing learning and performance (advanced); (c) ensure that assessment is aligned with instructional goals, anticipated learning outcomes, and instructional strategies (essential)” (Koszalka, Russ-Eft, and Reiser 2013, 54). This chapter provides additional information about this competency. It offers advice on how to identify the learning processes and outcomes to be measured, construct reliable and valid methods of assessing learning and performance, and ensure that assessment is aligned with instructional goals, anticipated learning outcomes, and instructional strategies.

Introduction Instructional designers should usually develop performance measurements during or immediately following the preparation of performance objectives. Measurements of all kinds—sometimes called metrics—have 215

216

Mastering the Instructional Design Process

been commanding attention in recent years. One reason has been growing demand by stockholders and stakeholders for accountability from all organizational levels. Another reason is that instructional designers are being held accountable for showing results for whatever training investments are made by their organizations. What Are Performance Measurements? Performance measurements are various means established by instructional designers for monitoring learner achievement. Paper-and-pencil tests are perhaps the most common. Test items may be developed directly from performance objectives before instructional materials are prepared. In this way, accountability for results is built into instruction early on. However, paper-and-pencil testing is not the only way to assess learner achievement. Other methods may also be used. For instance, trainees can be observed on the job as they perform the tasks they have learned. Computerized skills assessment is also becoming common, as is portfolio analysis, in which work samples are assessed. Why Are Performance Measurements Important? Performance measurements become benchmarks that, with performance objectives (discussed in the previous chapter), provide guidance to prepare instructional programs. They help answer an age-old question about every instructional experience: “What should be taught?” (Egan 1978, 72). They are important for three major reasons. First, they ensure economical choice of instructional content. Establishing performance measurements is part of the preliminary work to be completed before instructional materials are developed; it helps identify the content that should be included and the success level expected of learners upon completion of instruction. Second, performance measurements provide a basis for learner accountability to ensure that learner progress toward predetermined performance goals can be monitored during and after instruction. Third, performance measurements can help link up learner achievement to organizational strategic plans. Identifying What Learning Processes and Outcomes to Measure Instructional designers should be capable of developing tests, written questionnaires, interviews, and other methods of measuring performance.

Designing Learning Assessments

217

The performance measures should be written and correspond to performance objectives, rely on methods of measuring learning outcomes, comply with time and instructional constraints, and meet requirements for validity and reliability. Instructional designers should be able to develop performance measurements when furnished with necessary information on the characteristics of learners, the settings in which they are expected to perform, constraints on performance and instructional development, instructional objectives, and plans for analyzing needs and evaluating results. Stated simply, instructional designers should be able to answer two basic questions before they prepare instructional materials: (1) what should be measured? and (2) how should it be measured? To answer the first question, instructional designers should determine the purpose of the measurement and focus on methods of measuring instruction. To answer the second question, they should be able to design instruments—and write items for the instruments—to achieve the intended purpose. Deciding on the Purpose Once performance objectives have been written based on work requirements, instructional designers should decide: • What purpose will guide their performance measurement efforts? • What performance measurement methods should assess learners’ progress? • How performance should be measured. Instructional designers should always begin by clarifying their purposes for measuring performance. There are at least four possible purposes (Kirkpatrick and Kirkpatrick 2006): 1. Participant reaction. How much do participants enjoy what they are learning? How much do they enjoy the instructional methods used? 2. Participant learning. How well are participants meeting performance objectives? How well have they learned? 3. On-the-job performance change. How much change is evident on the job, based on what participants have learned? How well has the learning transferred from the instructional to the application environment? 4. Organizational impact. How has the organization been affected by the results of an instructional experience?

218

Mastering the Instructional Design Process

Determining Sources of Information After determining the purpose of performance measurement, instructional designers should next determine the sources of information used in measurement. There are three major sources of information. Performance objectives are the first. They should provide clues about what to measure because each objective must contain a measurable criterion for assessment. To measure performance instructional designers should consider how well learners have met the criterion set forth in each objective. Each objective should be directly tied to meeting job-related learning needs. Hence, measuring objectives provides information about how well learning needs are being met by instruction. Learner (worker) performance is the second source of information. Since instruction is—or should be—intended to improve individual performance in the workplace, information about what to measure should result from analysis of worker responsibilities, work standards, historical patterns of experienced workers’ performance problems on the job, and forecasts of likely future job changes. Using job descriptions, performance appraisal data, work standards, and such other information as emerges from the results of participant reaction sheets, instructional designers should be able to develop performance measures linked directly to successful job performance. Stakeholder preferences are the third source of information. Stakeholders are people who have a vested interest in instructional outcomes. Consider, for instance, what top managers and other interested parties want to know about instruction or its results. Often, instructional designers find that two key questions merit special consideration when measuring instruction or its results: (1) who wants to know? and (2) what do they want to know? A third question that may be addressed is, why do they want to know? Some instructional designers find it helpful to consult a menu of general questions about performance measures when deciding what to measure. Rae (1986, 9–10) developed such a menu, shown below, that still remains useful. Issue

Questions

Content of instruction

Is it relevant and in step with the instructional needs?

Method of instruction

Were the methods used the most appropriate ones for the subject?

Is it up-to-date?

Were the methods used the most appropriate for the learning styles of the participants?

Designing Learning Assessments

219

Issue

Questions

Amount of learning

What was the material of the course?

Instructor skills

Did the instructor have the attitude and skill to present the material in a way that encouraged learning?

Length and place of instruction

Given the material essential to learning, was the length and pace of the instruction adequate? Were some aspects of instruction labored and others skimped?

Objectives

Did the instruction satisfy its declared objectives?

Was it new to the learner? Was it useful, although not new to the learner, as confirmation or revision material?

Was the learner given the opportunity to satisfy any personal objectives? Was this need welcomed? Were personal objectives satisfied? Omissions

Were any essential aspects omitted from the learning event? Was any material included that was not essential to the learning?

Learning transfer

How much of the learning is likely to be put into action when the learner returns to work? If it is to be a limited amount only or none, why is this? What factors will deter or assist the transfer of learning?

Accommodation

If course accommodation is within the control of the instructor or relates to the instructional event, he or she may wish to ask whether the hotel or conference center training center was suitable. Was the accommodation acceptable? Were the meals satisfactory?

Relevance

Was this course/seminar/conference/workshop/ tutorial/coaching assignment/project the most appropriate means of presenting a learning opportunity?

220

Mastering the Instructional Design Process

Issue

Questions

Application of learning

Which aspects of your work now include elements which result directly from the learning event? Which new aspects of work have you introduced because of your learning? Which aspects of your previous work have you replaced or modified because of the learning? Which aspects of your learning have you not applied? Why not?

Efficiency

How much more efficient or effective are you in your work because of the instructional experience? Why or why not?

Hindsight

With the passage of time and attempts to apply the learning, are there any amendments you would wish to make to the training you received?

Select appropriate sources of information for performance measurement based on learner characteristics, setting resources and constraints, statements of performance objectives, and needs assessment or analysis or evaluation plan. Deciding How to Measure When deciding how to measure performance, instructional designers should apply the same classic criteria that Newstrom and Lilyquist (1979) have suggested in selecting a data collection method for needs assessment. The following five issues may warrant consideration: 1. Learner involvement: How much learner involvement is desired or feasible? 2. Management involvement: How much management involvement is desired or feasible? 3. Time required: How much time is available for measurement? 4. Cost: How much is the organization willing to spend to measure performance? 5. Relevant quantifiable data: How important is it for instructional designers to devise quantifiable measurements directly linked to on-the-job performance?

Designing Learning Assessments

221

Different methods of measuring performance earn high, moderate, or low ratings on each of these criteria. It is usually necessary to identify priorities—that is, determine which one is the most important, second most important, and so on.

An Overview of Steps in Preparing Instruments Having decided on a purpose (what is to be measured) and a measurement method (how it will be measured), instructional designers are then ready to develop measurement instruments. Instruments may be classified into three general types: (1) questionnaires, interview guides or schedules, observation forms, simulations, and checklists, (2) criterion-referenced tests, and (3) others. There are 10 basic steps to be taken during preparing a measurement instrument: 1. Clarifying the purpose of measurement and selecting an instrument. 2. Giving the instrument a descriptive title. 3. Conducting background research. 4. Drafting or modifying items. 5. Sequencing—or reviewing the sequence of—items. 6. Trying out the instrument on a small-group representative of the learner population. 7. Revising the instrument based on the small-group tryout. 8. Testing the instrument on a larger group. 9. Using the instrument—but establishing a means of tracking experience with it. 10. Revising the instrument—or items—periodically. These steps are summarized in the following paragraphs. Step 1: Clarifying the Purpose of Measurement and Selecting an Instrument Instructional designers should develop performance measurements by thinking through exactly why they are measuring instruction and, more important, what results they wish to achieve. Performance objectives are one starting point, since one purpose of measurement should usually be to determine how well learners have met instructional objectives by the end of the instructional experience. Instructional designers should ask

222

Mastering the Instructional Design Process

themselves, among other questions, this one: How can I find out whether these results are being achieved during the instructional experience and whether they were achieved following the instructional experience? At this point they can select or prepare an instrument well suited to helping answer this question. Step 2: Giving the Instrument a Descriptive Title If performance will be measured using an instrument developed by someone else, instructional designers should consider the title to see if it accurately describes what they wish to measure. If the instrument will be tailor-made, the title should be chosen with great care. The reason: by selecting a title, instructional designers focus their thinking on exactly what will be measured. Step 3: Conducting Background Research Instructional designers can often save themselves considerable time and effort by locating previously prepared instruments. One way to do that is to network with other instructional designers to find out whether they have developed instruments for similar purposes. In addition, instructional designers can sometimes successfully track down elusive instruments or research studies by using specialized reference guides. Tests in print can be located through the impressive library of the Educational Testing Service in Princeton, New Jersey, which maintains a collection of 10,000 tests. Background research on instrumentation will rarely be a complete waste of time. Even when instructional designers cannot locate instruments that measure exactly what they want, they may still locate examples that will stimulate new ideas about item layout or item sequence. When previously prepared instruments are found, instructional designers should decide whether to use them as they are or modify them to meet special needs. If previously prepared instruments can be easily modified, instructional designers can reduce the time and effort to prepare and validate an instrument. But if efforts to locate instruments or research are to no avail, then it will be necessary to prepare a tailor-made instrument. Begin instrument development by addressing several important questions: Who will be measured? Who will conduct the measurement? What will be measured? When will the measurement occur? Where will the measurement be conducted? How will the measurement be conducted?

Designing Learning Assessments

223

Step 4: Drafting or Modifying Items Relying on instructional objectives or other sources as a starting point, instructional designers should next decide what questions to ask to measure the changes wrought by the instructional experience. If a previously prepared instrument was located, each item must be reviewed to ensure that it is appropriate. Drafting original items or questions for interviews, questionnaires, observation forms, simulations, or checklists is a highly creative activity. Generate items or questions using focus groups or other creative methods. When drafting items, instructional designers should consider item format. Item format refers to the way performance is measured. Questionnaires or interview guides, for instance, may rely on open-ended items, closed-ended items, or some combination. Open-ended items produce qualitative or essay responses. The question “What do you feel you have learned in this instructional experience?” is an open-ended item. Closed-ended items produce quantifiable responses. Respondents asked to “rate how much you feel you learned during this instructional experience on a scale from 1 to 5, with 1 representing ‘very little’ and 5 representing ‘very much,’” are answering a closed-ended item. An instrument relies on a combination when it contains both open-ended and closed-ended items. Open-ended items are frequently used in conducting exploratory measurement studies. While the information they yield is difficult to quantify and analyze, they may also establish response categories for closed-ended instruments. Closed-ended items are frequently utilized in analytical measurement studies. Although the information they produce is easily quantified and analyzed, it can sometimes mislead if respondents are not given response categories. When that happens, respondents will select an approximation of what they believe and reply accordingly. Item format has a different, although related, meaning for observation forms, simulations, or checklists. These instruments are usually designed around observable behaviors associated with the instructional objectives or competent on-the-job performance. Instructional designers may prepare these instruments to count the frequencies of a behavior (How often did the learner do something?), assess the quality of a behavior (How well did the learner perform?), or both. The instrument user may exercise considerable flexibility in identifying what behavior to count or assess. Alternatively, the user may not exercise flexibility in assessing behaviors, because categories are predefined or methods of assessment have been provided on the instrument itself.

224

Mastering the Instructional Design Process

Item format has yet another meaning regarding tests. Developing criterion-referenced tests poses a challenge somewhat different from developing questionnaires, interviews, simulations, or other measurement instruments. Test preparation is an entire field of its own. When developing criterion-referenced tests, “the verb component of the instructional objective indicates the form that a test item should take” (Kemp 1985, 161). Examples of behaviors specified in instructional objectives and appropriately matched test item formats are shown in Table 13.1.

TABLE 13.1. BEHAVIORS SPECIFIED IN INSTRUCTIONAL OBJECTIVES AND CORRESPONDING TEST ITEMS

Type of test item

Brief description of test-item format

Behavior (verb specified in the instructional objective)

1. Essay (Example: “What are the chief advantages and disadvantages of the essay format as a test item?”)

A type of test item requiring a learner to respond in essay format. This type of item is appropriate for assessing higher levels of cognition—such as analysis, synthesis, and evaluation.

Construct Define Develop Discuss Generate Locate Solve State

2. Fill-in-the-blank (Example: “The________ in-the-blank is a type of test item.”)

A type of test item requiring the learner to fill in the blank with an appropriate word or phrase. Scoring can be objective because the required response is quite specific—often only one word is correct.

Construct Define Identify Locate Solve State

3. Completion (Example: “A type of test item that requires the completion of a sentence is called the__________ .”)

A type of test item that closely resembles the fill-in-the-blank type, except that the learner is asked to complete a sentence stem.

Construct Define Develop Discuss Generate Identify Locate Solve State

Designing Learning Assessments

225

TABLE 13.1. BEHAVIORS SPECIFIED IN INSTRUCTIONAL OBJECTIVES AND CORRESPONDING TEST ITEMS, cont’d.

Type of test item

Brief description of test-item format

Behavior (verb specified in the instructional objective)

4. Multiple-choice (Example: “A type of test item requiring the learner to choose from more than one possible answer is the (a) multiple-choice; (b) essay; (c) completion.”)

Kemp (1985, p. 162) calls multiple-choice “the most useful and versatile type of objective testing.” Learners must choose between three and five options or alternatives as the answer to a question.

Discriminate Identify Locate Select Solve

5. True-false (Example: “A true-false test item is less versatile than a multiple-choice one.” True-False)

A type of test item in which learners are asked to determine whether a statement is true or false.

Discriminate Locate Select Solve

6. Matching (See the example below.)

A type of test item in which learners are asked to match up items in one column with items in another column.

Discriminate Locate Select

For each item in column 1 below, select a corresponding item in column 2 by placing the number of the item before the item in column 1. Use items only once. Column 1 ————— 1. Essay ————— 2. Multiple-choice ————— 3. True-false 7. Project (Example: “Write an essay question to describe ten steps in preparing an assessment instrument.”)

Column 2 1. A type of test item in which learners have only two possible answers 2. A type of test item in which learners have between three and five alternatives 3. A test item requiring a narrative response A type of test in which learners are asked to demonstrate the ability to perform a task they have (presumably) learned through participation in an instructional experience.

Construct Develop Generate Locate Solve

Source: Taken from W. Rothwell and H. Kazanas, Mastering the Instructional Design Process: A Systematic Approach (4th ed.) (San Francisco: Pfeiffer, 2008), 201–202.

226

Mastering the Instructional Design Process

Step 5: Sequencing—or Reviewing the Sequence of—Items One choice is to sequence items in a logical order based on work tasks. Another choice is to sequence items according to a learning hierarchy. Step 6: Trying Out the Instrument on a Small-Group Representative of the Learner Population Sometimes called instrument pretesting, this step should not be confused with learner pretesting. If possible, instructional designers should select a sample of people representative of the learner population to participate in the instrument pre-test and ask for their help in identifying wording that is unclear or is otherwise inappropriate. Instructional designers should explain the instrument items to the group rather than ask them to answer the questions. Their responses should be noted for use during the next step. Step 7: Revising the Instrument Based on the Small-Group Tryout If a complete revision is necessary, which should rarely be the case, another small group should be selected for a second instrument pretest. Otherwise, instructional designers should revise items, based on their notes from the previous step, to improve clarity. Step 8: Testing the Instrument on a Larger Group The next step is a field test of the instrument on a larger group under conditions resembling, closely, those in which the instrument will later be used. The results of the field test should be noted. Step 9: Using the Instrument—But Establishing a Means of Tracking Experience with It Instructional designers should use the instrument but should also establish a way of tracking future experience with it. The results must be monitored. If tests are administered, instructional designers should periodically conduct item analysis to determine what questions the learners are missing and how often they are missing them. If questionnaires or interviews are used to measure performance, instructional designers must note the response patterns they receive to determine whether questions are yielding useful answers. If instructional designers are using structured observation, they should periodically review the categories they initially created.

Designing Learning Assessments

227

Step 10: Revising the Instrument—or Specific Items—Periodically As performance measurements are made using instruments, instructional designers gain experience. They can take advantage of that experience by periodically revising the instrument, or items on it. Revisions should also be made whenever changes are made to performance objectives or when new performance objectives are added. Other Methods of Measuring Performance Apart from questionnaires, interviews, simulations, and checklists, other methods may measure participant reactions, participant learning, on-thejob performance change, or organizational impact. However, not every method is appropriate for every purpose. These methods (note we do not call them items or instruments) include advisory committees, external assessment centers, attitude surveys, group discussions, exit interviews, and performance appraisals. An advisory committee is a group comprising stakeholders in instructional experiences. A committee may be established as standing (permanent and formal) or ad hoc (temporary and informal). One way to use an advisory committee is to ask its members to observe an instructional experience and assess how well they feel its objectives are achieved. Another way is to direct results of participant tests or other measures to committee members for interpretation. An external assessment center is measuring individual knowledge and skills. It is an extended simulation of job or group work. It could be used—although it would admittedly be expensive to do so—to determine what measurable change resulted from an instructional experience. An attitude survey is usually intended to assess individual perceptions about working conditions, coworkers, work tasks, and other issues. It could determine people’s perceptions of what changes or how much change resulted from instructional experiences. A group discussion is a meeting. It could identify measurement issues or assess a group’s perceptions about what changes or how much change occurred because of an instructional experience. An exit interview is a meeting with an employee just prior to the individual’s departure from an organization, department, or work unit. Sometimes, exit interviews may be combined with questionnaires mailed to terminating employees some time after they leave the organization. Exit interviews may identify measurement issues or assess an individual’s perceptions about what changes or how much change occurred because of an instructional experience.

228

Mastering the Instructional Design Process

A performance appraisal is an assessment of an individual’s job-related activities and results over a predetermined time frame. It could document a supervisor’s perceptions of what changes or how much change occurred because of an individual’s participation in an instructional experience.

Constructing Reliable and Valid Methods of Assessing Learning and Performance Three issues should be considered in any effort to assess or evaluate people. One is reliability; one is validity; and one is credibility. The Standards covers the first two. The third is also worthy of consideration because it is perhaps most practical of all. Reliability refers to the consistency of measures. It means that an assessment consistently measures what it is supposed to. There are several categories of reliability: • Interrater reliability examines how much agreement exists between several raters. • Test-retest reliability examines how much scores are consistent from one measurement opportunity to another. • Intermethod reliability examines how consistently scores are maintained when different approaches or instruments are used for the measurement. • Internal consistency reliability examines consistency in results across items in the same test. Validity refers to how accurately a measure responds to the real world. Can an assessment measure what it is supposed to measure? Many forms of validity exist. Among them: • Construct validity answers the question “how well does an assessment really measure what it is supposed to measure based on the theory?” • Convergent validity answers the question “how much is an assessment associated with other measures based on theory-based correlation?” • Discriminant validity answers the question “how well does a measure discriminate from other measures that are supposed to be unrelated?” • Content validity looks at the content of an assessment to answer the question “how representative is an assessment of behaviors it is supposed to measure?”

Designing Learning Assessments

229

• Representation validity looks at how well an assessment can be conducted. Is it practical and possible to measure? • Face validity refers to how well an assessment appears to measure what it is supposed to measure. • Criterion validity has to do with correlating an assessment and a criterion variable. How well does an assessment compare to other measures of the same thing? • Concurrent validity refers to how much the assessment correlates with other assessments of the same thing. • Predictive validity answers the questions “how much will an assessment predict the occurrence of the same thing?” and “will an assessment predict that something will happen?” Validity and reliability are important. Most forms of assessments must be tested for their reliability or validity, and that is especially important to do if employment decisions will be made because of a worker’s scores on an assessment. Often, competent statistical and psychometric consulting help must be used to ensure that all tests of validity and reliability are performed properly. That is true if assessment results may be challenged on the basis of employment discrimination. Credibility refers to the trustworthiness of the assessment or evaluation method or tool. Stated another way, do people such as managers believe in the assessment, and does a high or low score mean something that they believe? In many organizations, instructional designers may find that credibility with managers and workers is far more important for daily actions and decisions than statistical tests of reliability and validity.

Ensuring the Assessment Is Aligned with Instructional Goals, Anticipated Learning Outcomes, and Instructional Strategies To ensure that assessment is aligned with instructional goals, anticipated learning outcomes, and instructional strategies, instructional designers should develop simple yet elegant strategies to double-check what they have designed with: • Strategists • Learners • Other instructional designers on the team.

230

Mastering the Instructional Design Process

A good time to do that is during formative evaluation, assuming it is conducted. If it is not, then answering questions about these issues should be integrated with pilot tests or rapid prototyping and field-based testing efforts. It is as simple as asking stakeholders such questions as: • • • •

How well are assessments aligned with the organization’s strategic goals? How well are assessments aligned with the desired learning outcomes? How well are assessments aligned with instructional strategies? What can be done to improve alignment?

PART FOUR

EVALUATING AND IMPLEMENTING INSTRUCTIONAL DESIGN

Part VI: Predicting the Future of Instructional Design

Part I: Overview

Part V: Managing Instructional Design

Part II: Planning and Analyzing for Instructional Design

Part IV: Evaluating and Implementing Instructional Design

Part III: Designing and Developing for Instructional Design

CHAPTER FOURTEEN

EVALUATING INSTRUCTIONAL AND NONINSTRUCTIONAL INTERVENTIONS

I

n Chapter Two, we discussed the process for conducting a needs assessment, which helps the instructional design professional scope out a problem or opportunity that can be addressed by an instructional or noninstructional intervention. We then covered designing and utilizing learning assessments, identifying the desired outcomes and aligning these with the overall goals of the instructional intervention. These chapters provide the foundation for evaluation which, while conducted after the intervention, should be established early in the ID process when business and performance needs are identified, desired behavior changes are determined, and learning objectives and measurement strategies are set. This chapter summarizes the evaluation strategies and tactics used to answer the critically important question—what difference did it make? The answer to this question provides insights into potential improvements that can be made and valuable information to share with a variety of stakeholders who want to know if their investments are yielding worthy returns. In this chapter, we clarify assumptions about formative and summative evaluation, define key terms associated with these activities, and include a case study that dramatizes issues that can arise when developing a formative evaluation plan. We also describe the steps used to establish a formative evaluation plan and approaches to implementing that plan. This chapter 233

234

Mastering the Instructional Design Process

also describes summative evaluation and covers various post-intervention evaluation models.

Purpose and Definitions of Evaluation Instructional designers often hold the belief that their work is not finished until the targeted audience can learn from the material. The fourth edition of Mastering concentrated primarily on formative evaluation, which “involves gathering information on adequacy and using this information as a basis for further development” (Seels and Richey 1994, 57). This ensures that the instruction is sound before it is deployed to large numbers of intended learners. This edition provides more complete coverage of summative evaluation, which “involves gathering information on adequacy and using this information to make decisions about utilization” (Seels and Richey 1994, 57). It helps to determine the results of the instruction post-implementation. Evaluation in all its forms has figured prominently in instructional design practice as decision makers demand increasing accountability (Rothwell, Lindhold, and Wallick 2003). According to The Standards (Koszalka, Russ-Eft, and Reiser 2013, 55), even though the competency and two performance statements associated with evaluation are classified as advanced, “even a novice instructional designer should be familiar with the need for evaluation and revision prior to any dissemination of instructional products and programs.” High accountability is now being demanded for all or most forms of instruction and evaluation skills are critical for all designers to possess. As organizations have tightened their budgets, the need to prove that training is a worthwhile investment has become more and more important. The performance statements associated with this competency indicate that instructional designers should be able to (Koszalka, Russ-Eft, and Reiser 2013, 56): “(a) Design evaluation plans (advanced); (b) Implement formative evaluation plans (essential); (c) Implement summative evaluation plans (essential); and (d) Prepare and disseminate evaluation report (advanced).” Stakeholders in Evaluation Stakeholders are those who are interested in the results of an evaluation effort and can range from front line employees to senior level executives including the CEO (Phillips and Phillips 2010a; Phillips and Phillips, 2010b). The interests and objectives of stakeholders can vary and can even

Evaluating Instructional and Noninstructional Interventions

235

TABLE 14.1. INTEREST IN EVALUATION BY STAKEHOLDER Stakeholder

Key Questions or Interests

Instructional designers

Did the learning work as intended? Did learners respond favorably to the learning? What improvements can be made for next time? Were learners satisfied with the experience? Did learners achieve the intended outcomes? How can I improve my delivery? Did my knowledge or skill improve? Am I more productive or effective in my job? Did my people acquire new knowledge or skills? Are they more productive or effective in their jobs? Did the benefits received outweigh the “cost” of participating? Are learners more productive? Are learners demonstrating behaviors that will further our strategic objectives? Are learners adding greater value than prior to participating?

Instructors or facilitators

Learners Managers of learners

Executives or sponsors

compete. Table 14.1 lists stakeholders and their potential interest(s) in the outcomes of the evaluation. Data Collection Approaches When an instructional designer engages in evaluation, it is similar to a scientist conducting research. Many of the same principles, strategies, and techniques apply. Research has many purposes, but “there are two basic purposes for research: to learn something, or to gather evidence” (Taflinger 2011). Like the researcher, the instructional designer involved in evaluation efforts works to frame important questions and systematically attempts to answer those questions. The ability to collect and analyze data is central to the role of the evaluator. Sources of Evaluation Data There are many data sources that can be tapped into for evaluation. The data may be quantitative or numeric, such as survey responses using

236

Mastering the Instructional Design Process

a 5-point scale. Qualitative data is non-numeric like the results of an interview. Some of the most commonly utilized sources of data include: the learners themselves, managers, subject experts, program sponsors, and what Allison Rossett (1987, 25) refers to as extant data or the “stuff that companies collect that represents the results of employee performance.” Examples of extant data include organizational performance data or reports (sales, profitability, productivity, quality), engagement or satisfaction surveys, and talent data or reports (retention, performance ratings, promotion, mobility, exit interviews). Use of a single data source can lead to inaccurate evaluation results because the perspective being considered may be highly subjective or limited. It is therefore important to use multiple data sources whenever feasible and practical. Doing so can increase the validity of the data, a subject addressed later in this chapter. Data Collection Methods. Similar to the recommendation of tapping into

multiple data sources when conducting an evaluation, we also recommend using multiple data collection methods when possible. Methods are how data are collected from source(s). Multiple sources and methods will help to ensure the designer does not have gaps in the data collected, thereby providing more valid information upon which to make decisions. Below are several data collection methods. Interviews. The interview is a commonly used method to gather feedback and input directly from key sources. To ensure consistency of approach, interview protocols may be developed to articulate questions to be posed to the interviewee. Some of the advantages of the interview method include directly hearing responses from the interviewee, the ability to observe body language, and the opportunity to pose follow up questions to dig deeper into a response. It’s helpful when interviews are recorded to aid with transcription and later analysis, but this has the potential downside of causing discomfort for the interviewee. Interviews yield qualitative data, sometimes voluminous amounts, which require skill and expertise to analyze effectively. Focus Groups. The focus group is similar to the interview because it is typically conducted in-person or through virtual technologies. Unlike an interview often conducted with one individual, a focus group involves multiple people (8–12 people is common). It’s an effective way to gain multiple perspectives relatively quickly and on a wide range of topics or questions of interest. Having a skilled facilitator helps keep the conversation moving,

Evaluating Instructional and Noninstructional Interventions

237

ensures full participation, and avoids groupthink (when individual voices, especially those who may have dissenting opinions, are withheld to conform to the will of the group—which is often falsely reflected by who talks the most or the loudest). A skilled facilitator will be on the watch for groupthink and other dysfunctional behaviors and take actions to prevent or deal effectively with them. Sometimes a “scribe” is used to capture key points and responses from the focus group or it may be recorded and later transcribed. Observation. Observing someone in their work environment is a labor-intensive data collection method whereby a trained observer watches learners or performers at work to determine if behaviors, performance, interactions, or other variables have changed or improved. Obtrusive observations, where the observer is present and visible to the performer or subject, provide a direct and first-hand means by which to gather data and ask questions during the observation. Unobtrusive observation methods are when the presence of the observer is not known or highly visible to the research subject, thus creating a more natural environment where there is less likelihood of the Hawthorne Effect (when behavior is changed by the mere presence of extra attention being paid). Quantifying observational data is sometimes aided with tools like checklists, flow charting tools, or frequency counts. Surveys or Questionnaires. These are data collection methods that can yield quantitative and qualitative responses depending on the intended outcomes and research design used. Questions with rating scales yield numeric data that can be analyzed, while open-ended questions allow the respondent to provide narrative comments. Issues of validity and reliability make possessing or having access to expertise in the design of surveys important. In the past, surveys were mainly conducted using paper and pencil, but today many software packages and online tools such as Survey Monkey, Zoomerang, and Poll Everywhere make the mass collection of data more efficient. Many even come with basic analysis and reporting tools, and also provide raw data that can be imported into more sophisticated data analytic software such as SPSS or Excel. An example of a survey that contains qualitative and quantitative questions is shown in Exhibit 14.1. Tests. Tests can include knowledge tests used to assess acquisition of information, such as the essential elements of a company’s balance sheet or detailed knowledge of the organization’s products. Skill tests are used to assess whether behaviors, such as conducting an effective performance review, have improved. Sometimes evaluators use pretests to determine knowledge or skills prior to the learning intervention to establish a

238

Mastering the Instructional Design Process

EXHIBIT 14.1. SAMPLE QUESTIONNAIRE WITH QUANTITATIVE AND QUALITATIVE DATA Directions: Please answer the following questions about how you feel you are working together as a team. Scale 1 = Strongly disagree, 2 = Somewhat disagree, 3 = Neither agree or disagree, 4 = Somewhat agree, 5 = Strongly agree 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

We demonstrate respect for one another. We regularly give each other open and honest feedback. We value each others’ contributions. We put the results of the organization ahead of our own team’s results. Team meetings are productive. We are completely committed to decisions that we agreed upon. We are able to discuss conflicts with one another. We are all aligned with the overall goals of the organization. We have trust for one another. We assume positive intent with one another.

Directions: Please answer the following questions: 1. What are the three ways in which we work best as a team?

2. What would you describe as the one thing we could be doing differently to become a better team?

baseline. Post-tests are used following learning to measure the acquisition of knowledge or skill, ideally compared to performance on the pretest. Criterion-referenced tests (CRT) establish objective standards of performance typically clarified to learners so they visualize success and eventually attain it through practice, feedback, and eventual mastery. Norm-reference tests (NRT) compare performance against others who have taken the test rather than against an established standard of performance. Similar to surveys, the ability to construct tests and write effective questions is a specialized skill where deep subject expertise is important to ensure validity and reliability.

Evaluating Instructional and Noninstructional Interventions

239

Extant Data Review. It’s helpful to leverage organizational data that already exists and use it for evaluation purposes. Sometimes extant data is easy to obtain, such as when organizational records are well-kept and easily accessible in a weekly sales report or quarterly review of customer survey results. In other cases, the evaluator may need to comb through paper files to pull together and make sense of important but poorly maintained and sometimes inaccurate data. There are often numerous potential extant data that could be leveraged for evaluation. The challenge becomes narrowing the scope to the data that will be most valuable to the key stakeholders, data that’s believed to be accurate and valid, and data that’s easy to access and analyze.

Formative Evaluation Evaluation that happens at the end of an intervention is summative evaluation. However, the intervention should also be evaluated throughout the design and development process. This evaluation is called formative evaluation and it helps to pinpoint adjustments that must be made during the design process, so desired results are more likely to be achieved. Assumptions about Formative Evaluation Instructional designers make three fundamental assumptions when evaluating instructional materials and methods. First, they view evaluation as primarily a formative process. This assumption rests on the belief that instructional materials and methods should be evaluated—and revised— prior to widespread use to increase their instructional effectiveness. In this way, it is hoped that learner confusion will be minimized. Second, instructional designers assume that evaluation means placing value on something. Evaluation is not objective and empirical; rather, it rests heavily on human judgment and human decisions. Human judgment reflects the individual values of instructional designers and the groups they serve. Third, instructional designers expect to collect and analyze data as part of the evaluation process. To determine how well instructional materials and methods work, instructional designers must try them out. It is then possible, based on actual experience with learners, to make useful revisions to the materials.

240

Mastering the Instructional Design Process

Defining Terms Associated with Formative Evaluation Before undertaking a formative evaluation, instructional designers should take the time to familiarize themselves with at least two key terms: formative product evaluation and formative process evaluation. However, instructional designers should also minimize the use of this special terminology. Operating managers or clients will only be confused or turned off by it. Formative Product Evaluation. The term formative product evaluation means appraising instructional materials during preparation. Its key purposes are to provide instructional designers with descriptive and judgmental information about the value of instruction. Descriptive information outlines the value of instructional components. Judgmental information assesses how much learning results from the instructional materials when used with learners and places a value on those results. Formative Process Evaluation. Formative process evaluation is related to formative product evaluation and means the appraisal of instructional methods, including how planned learning experiences are delivered or facilitated. Like product evaluation, it provides both descriptive and judgmental information about planned learning experiences.

Developing a Formative Evaluation Plan Instructional designers should develop a formative evaluation plan that focuses attention on the instructional materials. There are seven steps in developing a formative evaluation plan. We will describe them in the following sections. Step 1: Determining Purpose, Objectives, Audience, and Subject The first step of formative evaluation is to determine the purpose, objectives, audience, and subject. Answer the question, why is this evaluation being conducted? How much is the focus solely on the quality of the instructional materials or methods, and how much is it on other issues, such as the following (Kirkpatrick 1996): • How much will the targeted learners enjoy the instructional materials, content, or methods? • How much will the participants learn?

Evaluating Instructional and Noninstructional Interventions

241

• How much impact will the learning experience have on the participants’ job performance? • How much impact will the planned learning experience have on the organization? As part of the first step, clarify the desired results of the formative evaluation. For each purpose identified, establish measurable objectives for the evaluation. In this way, instructional designers help themselves and others assess the results against what was intended. In addition, consider who wants the evaluation and why. Is it being conducted primarily for the benefit of instructional designers, senior leaders, key decision makers, immediate supervisors of the targeted learners, or some combination of all these groups? Always clarify who will review the results of the formative evaluation and what information they need from it. This will help to identify what to evaluate and how to present the findings. Identify who will participate in the formative evaluation. Will the evaluation be focused on representative targeted learners only, or will it also focus on learners with special needs or low abilities? Subject-matter specialists? Representatives of the supervisors of targeted trainees? Their managers? Senior leaders? There are reasons to target formative evaluation to each group of subjects, depending on the purpose and objectives of the evaluation. Step 2: Assessing Information Needs The second step in conducting formative evaluation is to assess the information needs of the targeted audiences. Precisely what information is sought from the results of the formative evaluation? Usually, the targeted audiences will provide important clues about information needs: • Instructional designers will usually be interested in how they can revise instructional materials or delivery methods to make them more effective for learners. • Key decision makers will usually be interested in how well the materials meet previously identified instructional needs and solve human performance problems. They may also want to assess how much and what kind of financial or managerial support is necessary to ensure instructional success or on-the-job application of what was learned. • Immediate supervisors of targeted learners will usually be interested in familiarizing themselves with the instructional content so they can hold learners accountable on their jobs for applying what they learned.

242

Mastering the Instructional Design Process

• Representatives of the targeted learners may be interested in how easy or difficult the instructional materials are and how test results will be used. In addition, consider the extent to which each group might be interested in determining how well instructional materials and methods convey the content, allow participants to apply what they learn, measure accomplishment, and demonstrate learner achievement of performance objectives. Step 3: Considering Proper Protocol The third step in conducting a formative evaluation is to consider proper protocol. Several questions about the protocol of conducting formative evaluation should be answered: • How much do the targeted audiences expect to be consulted about a formative evaluation before, during, and after it is conducted? • What permissions are necessary to carry out the study? • Whose permissions are necessary? • What formal or informal steps are necessary to secure the permissions to conduct a formative evaluation, select subjects, collect data, and feed back results? Protocol is affected by five key factors: (1) the decision makers’ experience with formative evaluation, (2) labels, (3) timing, (4) participation, and (5) method of evaluation. The decision makers’ experience with formative evaluation is the first factor influencing protocol. If the decision makers have had no experience with formative evaluation, instructional designers should take special care to lay the foundation for it by describing to the key stakeholders what it is and why it is necessary. If decision makers have had experience with formative evaluation, determine what mistakes (if any) were made in previous evaluative efforts so repeating them can be avoided. Common mistakes may include forgetting to secure the Evaluating Instruction permissions, forgetting to feed back to decision makers information about evaluation results, and forgetting to use the results in a visible way to demonstrate that the evaluation was worth the time and effort. Labels are a second factor affecting protocol. Avoid using the imposing term “formative evaluation” with anyone other than instructional designers, since it may only create confusion. Try more descriptive labels such as walkthroughs, rehearsals, tryouts, or executive previews.

Evaluating Instructional and Noninstructional Interventions

243

Timing is a third factor affecting protocol. Is it better to conduct a formative evaluation at certain times in the month or year than at other times, due to predictable work cycles or work schedules? Make sure that formative evaluations will not be carried out when they conflict with peak workloads or other events, like a company board meeting or an earning’s call which may make it difficult for key stakeholders to approve or participate. The participation of key stakeholders is a fourth factor affecting protocol. How essential is it to obtain permission from a few key individuals before conducting a formative evaluation? If essential, who are they? How is their permission secured? How much time should be allowed for obtaining the permissions? The method of evaluation is the fifth and final factor affecting protocol. Given the organization’s culture, should some instruments, methods of data collection, or analysis be used instead of others? Instructional designers should never underestimate the importance of protocol. If protocol is forgotten, instructional designers can lose support for the instructional effort before it begins. Remember, any instructional experience is a change effort. Also, formative evaluation, like needs assessment, offers a valuable opportunity to build support for change. But if proper protocol is violated, it could work against success. The audiences will focus attention on the violation, not instructional materials or methods. Step 4: Describing the Population to Be Studied and Selecting the Subjects The fourth step in conducting formative evaluation is to describe the population for study and to select participants. Always describe from the outset the population to be studied. Usually, instructional materials or methods should be tried out with a sample, usually chosen at random, from the targeted group of learners. But take care to precisely clarify the learners with whom the materials will be used. Should participants in formative evaluation be chosen on the basis of any specialized information such as situation-related characteristics, decision-related characteristics, or learner related characteristics? Sometimes, it may be appropriate to try out instructional materials or methods with such specialized populations as exemplars (the top performers), veterans (the most experienced), problem performers (the lowest performers), novices (the least experienced), high-potential workers (those with great, but as yet unrealized, performance capabilities),

244

Mastering the Instructional Design Process

or disabled workers. Formative evaluations conducted with each group will yield specialized information about how to adapt instructional materials to account for unique needs rather than taking a one-size-fits-all approach. Once the learners have been identified, select a random sample. Use automated human resource information systems for that chore. If a specialized population is sought for the study, other methods of selecting a sample may be substituted. These could include announcements to employees or supervisors, word-of-mouth contact with supervisors, or appeals to unique representatives. If specialized methods of selecting participants for formative evaluation are used, consider the protocol involved in contacting possible participants, gaining their cooperation, securing permission from their immediate supervisors or union representatives, and getting approval for any time off the job that may be necessary. Step 5: Identifying Other Variables of Importance The fifth step in conducting a formative evaluation is to identify other variables of importance. Ask these questions to identify the variables: 1. What settings should be used for the formative evaluation? 2. What program issues are particularly worth pretesting before widespread delivery of instruction? 3. Should the formative evaluation focus solely on instructional issues, and how much (if at all) should it focus on other important but noninstructional issues such as equipment needs, staff needs, financial resources required, facility requirements, and noninstructional needs of participants? 4. What positive but postinstructional outcomes of the planned learning experience can be anticipated? What negative postinstructional outcomes can be anticipated? 5. What estimates should be made about expected costs of the instructional program? 6. How accurate are the prerequisites previously identified? Step 6: Formulating a Study Design The sixth step in conducting a formative evaluation is to create an evaluation design. The central question is this: How should the formative evaluation be conducted?

Evaluating Instructional and Noninstructional Interventions

245

An evaluation design is comparable, in many respects, to a research design (Campbell and Stanley 1966), except that its purpose is to judge instructional materials and methods rather than make new discoveries. An evaluation design is the “plan of attack”—the approach to be used in carrying out the evaluation. In formulating a design, be sure to (1) define key terms; (2) clarify the purpose and objectives of the evaluation; (3) provide a logical structure or series of procedures for assessing instructional materials and methods; (4) identify the evaluation’s methodologies, such as surveys, trial runs or rehearsals, and interviews; (5) identify populations to be studied and means by which representative subjects will be selected; and (6) summarize key standards by which the instructional materials and methods will be judged. Step 7: Formulating a Management Plan to Guide the Study The seventh and final step in conducting a formative evaluation is to formulate a management plan, a detailed schedule of procedures, events, and tasks to be completed to implement the evaluation design. A management plan should specify due dates and descriptions of the tangible products resulting from the evaluation. It should also clarify how information will be collected, analyzed, and interpreted in the evaluation. The importance of a management plan should be obvious. When a team is conducting a formative evaluation, the efforts of team members must be coordinated. A management plan helps avoid the frustration that results when team members are unsure of what must be done, who will perform each step, and where and when the steps will be performed. There are two ways to establish a management plan. One way is to prepare a complete list of the tasks to be performed, preferably in the sequence in which they are to be performed. This list should be complete and detailed, since this task-by-task management plan becomes the basis for dividing up the work of instructional designers, establishing timetables and deadlines, holding staff members accountable for their segments of project work, and (later) assessing individual and team effort. A second way is to describe the final work product of the project and the final conditions existing on project completion. What should the final project report contain? Who will read it? What will happen because of it? How much and what kind of support will exist in the organization to facilitate the successful introduction of the solution? Ask team members to explore these and similar questions before the formative evaluation

246

Mastering the Instructional Design Process

plan is finalized, using their answers to organize the steps to achieve the final results.

Four Major Approaches to Conducting Formative Evaluation Although there are many ways to conduct formative evaluation (Bachman 1987; Chernick 1992; Chinien and Boutin 1994; Dick and King 1994; Gillies 1991; Heideman 1993; Russell and Blake 1988; Tessmer 1994; Thiagarajan 1991), four major approaches will be discussed here. Each has its own unique advantages and disadvantages. These approaches may be used separately or in combination. These include: 1. 2. 3. 4.

Expert reviews. Management or executive rehearsals. Individualized pretests and pilot tests. Group pretests and pilot tests. We will describe each approach briefly.

Expert Reviews There are two kinds of expert reviews: (1) those focusing on the content of instruction and (2) those focusing on delivery methods. Most instructional designers associate expert reviews with content evaluation. Expert reviews focusing on content are, by definition, conducted by subject-matter experts (SMEs), individuals whose education or experience regarding the instructional content cannot be disputed. Expert reviews ensure that the instructional package, often prepared by instructional designers experts (IDEs) who may not be versed in the specialized subject, follows current or desired work methods or state-of-the-art thinking on the subject. A key advantage of the expert review is that it ensures that materials are current, accurate, and credible. Expert reviews may be difficult and expensive to conduct if “experts” on the subject cannot be readily located or accessed. Begin an expert review by identifying experts from inside or outside the organization. Do that by accessing automated human resource information systems (skill inventories) if available, contacting key management personnel, or conducting surveys. Identify experts outside the organization by asking colleagues, accessing automated sources such as the

Evaluating Instructional and Noninstructional Interventions

247

Association for Talent Development’s Membership Information Service, or compiling a bibliography of recent printed works on the subject and then contacting authors. Once the experts have been identified, prepare a list of specific, open-ended questions for them to address about the instructional materials. Prepare a checklist in advance to ensure that all questions you want answers to are considered and answered thoroughly. See Exhibit 14.2. Expert reviews are rarely conducted in group settings; rather, each expert prepares an independent review. The results are then compiled and used by instructional designers to revise instructional materials.

EXHIBIT 14.2. CHECKLIST FOR SME REVIEW OF INSTRUCTIONAL MATERIALS Name of the SME: _____________________________________ Instructional Material

Questions

1. Audiovisuals

Do the audiovisuals represent the content well? Are they clear? Do these visuals enhance learning? What would you change about them? Is this the best way to present the material? Do the course materials assist in the learning process? Are they clear and easy to read/use? What would you change? Do the content assist in the learning process? Is the content easy to understand? Does the content help you achieve your learning goals? What would you change? Is the amount or length of content appropriate? Are the examples and case studies appropriate? Do they help you to understand the content? Would you change or use different case studies or examples?

2. Course materials

3. Content

4. Case studies, examples, etc.

Feedback

248

Mastering the Instructional Design Process

Expert reviews that focus on delivery methods are sometimes more difficult to conduct than expert reviews focusing on content. The reason: experts on delivery methods are not that easy to find. One good approach is to ask “fresh” instructional designers, those who have not previously worked on the project, to review instructional materials for the delivery methods used. For each problematic issue the reviewers identify, ask them to note its location in the instructional materials and suggest revisions. Another good approach is to ask experienced instructors or tutors to review an instructional package. If the package is designed for group-paced, instructor-led delivery, offer a dress rehearsal and invite experienced instructors to evaluate it. If the package is designed for individualized, learner-paced delivery, ask an experienced tutor to try out the material. Management or Executive Rehearsals Management or executive rehearsals differ from expert reviews. They build support by involving key stakeholders in the preparation and review of instructional materials prior to widespread delivery. In a management rehearsal, an experienced instructor describes to supervisors and managers of the targeted learners what content is covered by the instructional materials and how they are to be delivered. No attempt is made to “train” the participants in the rehearsal; rather, the focus is on familiarizing them with its contents so they can provide support to and hold their employees accountable for on-the-job application. To conduct a management or executive rehearsal, begin by identifying and inviting key managers to a briefing of the materials. Some instructional designers prefer to limit invitations to job categories, such as top managers or middle managers. Others prefer to offer several with various participants rehearsals. Prepare a special agenda for the rehearsal. Make it a point to cover at least the following eight aspects: (1) the purpose of the instructional materials; (2) the performance objectives; (3) the business needs, human performance problems, challenges, or issues addressed by the instruction; (4) a description of targeted learners; (5) evidence of need; (6) an overview of the instructional materials; (7) steps taken so far to improve the instruction; and (8) steps that members of this audience can take to encourage application of the learning in the workplace.

Evaluating Instructional and Noninstructional Interventions

249

Individualized Pretests and Pilot Tests Individualized pretests, conducted onsite or offsite, is another approach to formative evaluation. Frequently recommended as a starting point for trying out and improving draft instructional materials, they focus on learners’ responses to instructional materials and methods, rather than those of experts or managers. Most appropriate for individualized instructional materials, they are useful because they yield valuable information about how well the materials will work with the targeted learners. However, pretests and pilot tests have their drawbacks: they can be time consuming, and they require learners to take time away from work and may pose difficulties for supervisors and co-workers in today’s lean staffed, right-sized organizations. Individualized pretests are intensive “tryouts” of instructional materials by one learner. They are conducted to find out just how well one participant fares with the instructional materials. A pretest is usually held in a nonthreatening or off-the-job environment, such as in a corporate training classroom or learning center. Instructional designers should meet with one person chosen randomly from a sample of the target population. Begin the session by explaining that the purpose of the pretest is not to “train” or evaluate the participant but, instead, to test the material. Then deliver the material one-on-one. Each time the participant encounters difficulty, encourage the person to stop and point it out. Note these instances for future revision. Typically, instructional designers should direct their attention to the following three issues: (1) How much does the participant like the material? (2) How much does the participant learn (as measured by tests)? (3) What concerns does the participant express about applying what he or she has learned on the job? Use the notes from this pretest to revise the instructional materials. The individualized pilot test is another approach to formative evaluation. It is usually conducted after the pretest, and focuses on participants’ reactions to instructional materials in a setting comparable to that in which the instruction is to be delivered. Like pretests, pilot tests provide instructional designers with valuable information about how well the instructional materials work with representatives from the group of targeted learners. However, their drawbacks are similar to those for pretests: they can be time consuming, and they require learners to take time away from work.

250

Mastering the Instructional Design Process

Conduct a pilot test in a field setting, one resembling the environment in which the instructional materials are used. Proceed exactly as for a pretest with the following six steps: (1) select one person at random from a sample of the target population; (2) begin by explaining that the purpose of the pilot test is not to train or evaluate the participant but to test the material; (3) progress through the material with the participant in a one-to-one delivery method; (4) note each instance in which the participant encounters difficulty with the material; (5) focus attention on how much the participant likes the material, how much the participant learns as measured by tests, and what concerns the participant raises about applying on the job what he or she has learned; and (6) use the notes from the pilot test to revise instructional materials prior to widespread use. Group Pretests and Pilot Tests Group pretests resemble individualized pretests but are used to try out group-paced, instructor-led instructional materials. Their purpose is to find out just how well a randomly selected group of participants from the targeted learner group fares with the instructional materials. Held in an off-the-job environment, such as in a corporate training classroom or learning center, the group pretest is handled precisely the same way as an individualized pretest. A group pilot test resembles an individualized pilot test but is delivered to a group of learners from the targeted audience, not to one person at a time. Typically the next step following a group pretest, it focuses on participants’ reactions to instructional materials in a field setting, just like its individualized counterpart. Administer attitude surveys to the learners about the experience, and written, computerized assessments, or demonstration tests to measure learning. Realize in this process that a relationship exists between attitudes about instruction and subsequent on-the-job application (Dixon 1990). Using Approaches to Formative Evaluation Each approach to formative evaluation is appropriate under certain conditions. Use an expert review to double-check the instructional content and the recommended delivery methods. Use a management or executive rehearsal to build support for instruction, familiarize key stakeholders with its contents, and establish a basis for holding learners accountable

Evaluating Instructional and Noninstructional Interventions

251

on the job for what they learned off the job. Use individualized pretests and pilot tests to gain experience with, and improve, individualized instructional materials prior to widespread delivery; use group pretests and pilot tests to serve the same purpose in group-paced, instructor-led learning experiences.

Providing Feedback from Formative Evaluation One final issue to consider when conducting formative evaluation is how to provide feedback to key stakeholders about the study and its results. The shorter the report, the better. One good format is to prepare a formal report with an attached, and much shorter, executive summary to make it easier for the reader, a one to two page. The report should usually describe the study’s purpose, key objectives, limitations, and any special issues. It should also detail the study methodology (including methods of sample selection) and instruments prepared and used during the study, and should summarize the results. Include copies of the instructional materials reviewed, or at least summaries. Then describe the study’s results, including descriptions of how well learners liked the material, how much they learned as measured by tests, what barriers to on-the-job application of the instruction they identified, and what revisions will be made to the materials. Formative product evaluation results are rarely presented to management, since their primary purpose is to guide instructional designers in improving instructional materials. However, instructional designers can feed back the results of formative evaluation to management as a way of encouraging management to hold employees accountable on the job for what they learned.

Summative Evaluations Summative evaluation involves gathering information about a learning intervention after it has been deployed. It helps the instructional designer and other key decision makers identify what worked and what didn’t work, determine value, and report on the difference made because of the solution. Besides identifying improvements, summative evaluation also helps to determine next steps such as accelerating the deployment to reach learners more quickly, expanding deployment to reach more

252

Mastering the Instructional Design Process

learners, and sometimes discontinuing the intervention if results are deemed insufficient relative to the costs.

Kirkpatrick’s Four Levels In 1960, Donald Kirkpatrick introduced his now famous “Four Levels” framework (Kirkpatrick 1959, 1960). Still today, this is the most widely used framework for thinking about and conducting learning evaluation in organizations. Level 1 focuses on learner satisfaction, level 2 evaluates acquisition of new knowledge or skill, level 3 examines learning transfer from the classroom to the workplace, and finally, level 4 determines the impact of the intervention on organizational or business outcomes. With each successive “level” of evaluation, starting with level 1, the focus moves from the individual to the organizational impact of the intervention. Each level yields different insights and is important for various reasons. Rigor, resource intensity, sophistication and expense also increases with each successive level. The frequency with which the levels are employed within organizations decreases as you ascend from level 1 through level 4. Various stakeholders place greater or lesser importance on the different levels. The time at which each level of evaluation is used also differs. Levels 1 and 2 occur during or immediately after the intervention is complete whereas Levels 3 and 4 evaluations can be conducted days, months, or even years after the invention. This section will detail each of the four levels and some of the key applications and considerations to apply them effectively. Level 1. This level of evaluation attempts to measure learner satisfaction (“did they like it?”). This form of evaluation is used most frequently in organizations. The most common means of gathering level 1 data is via a post-delivery survey, which is often and somewhat sarcastically referred to as a “smile sheet,” not to minimize the value of level 1 evaluations—they are often the lifeblood of professional facilitators. Many instructors can barely stand to wait to receive their evaluation results after delivery is complete. This is because it provides almost instantaneous feedback about how well (or poorly) they did during the session. Facilitators can gain valuable insights into their performances and training managers can hold facilitators accountable through level 1 feedback. Facilitator effectiveness is only one of several dimensions that can be assessed through this form of evaluation. Facilities and room set-up, preprogram communications and logistics, and food quality and service are important insights for training managers and operational staff. Instructional designers are highly interested

Evaluating Instructional and Noninstructional Interventions

253

in level 1 data as it relates to attainment of instructional/performance objectives, participant material, activities and exercises, content, length, and flow. Data can be collected on all via a level 1 evaluation. Level 1 evaluations are relatively easy to administer and yield many useful insights. In the past, administration was typically via a paperbased survey that learners completed at the end of the program. Today, paper-based evaluations may still be used, but in many organizations they have been replaced by online surveys using commonly available survey software applications such as Zoomerang or Survey Monkey, or more powerful integrated software platforms like Metrics That Matter (www .knowledgeadvisors.com). A recent development in level 1 evaluation is to incorporate the concept of Net Promoter Score (NPS) (see www.netpromoter.com/whynet-promoter/know). NPS is a marketing and customer loyalty concept pioneered by Fred Reichheld (2006), who states that NPS “is based on the fundamental perspective that every company’s customers can be divided into three categories: Promoters are loyal enthusiasts who keep buying from a company and urge their friends to do the same. Passives are satisfied but unenthusiastic customers who can be easily wooed by the competition. Detractors are unhappy customers trapped in a bad relationship” (19). One question is asked about the respondent’s willingness to refer or recommend this company, product, or service to a colleague or friend. A calculation is made based on the number of respondents who fall into each category and an NPS score is calculated and can be tracked. Besides the primary quantitative question, respondents are also asked to provide qualitative feedback stating the reason for the score and what the company can do to improve. Mattox (2013) has conducted research and introduced strategies and approaches to incorporating NPS into learning evaluation. While NPS is growing in popularity, it is not without its critics and skeptics (Thalheimer 2013). Despite their widespread use and obvious benefits, level 1 evaluations have limitations. Level 1 feedback is highly subjective because it is reporting how the learners felt about the experience. Learner satisfaction, like customer satisfaction, is of critical importance, but a long running fable in the learning field is of a study that found there was a stronger correlation between learner satisfaction and the size of the donuts served in the morning of the program, than any other variable—the larger the donuts, the higher the level 1 evaluation scores! (See Exhibit 14.3 for a sample level 1 evaluation.)

254

Mastering the Instructional Design Process

EXHIBIT 14.3. LEVEL 1 EVALUATION FORM Directions: To evaluate your satisfaction with this program, please circle the number that most closely represents your opinion. Neither Completely Somewhat agree nor Somewhat Strongly disagree disagree disagree agree agree Overall, this program was helpful The materials used in this program helped me to understand the content The type of instruction helped me to learn the principles and concepts of this program The location was appropriate Participating with the other learners was useful to me This program took about the right amount of time needed I would recommend this program to my friends and colleagues I am likely to take other instructional programs similar to this one

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1 1

2 2

3 3

4 4

5 5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

Please add anything else that you would like to say about this program:

Level 2. The purpose of Level 2 evaluations is to ascertain knowledge or skill acquisition (“did they learn it?”). While level 1 evaluation is used the most frequently, level 2 evaluation is also common. Testing is the most common form of level 2 evaluation. Knowledge tests attempt to evaluate a learner’s increase in awareness or understanding of a new concept, idea, or information. They are often used at the end of a learning program, but can also be embedded throughout. Sometimes a pretest is administered to determine baseline knowledge before learning begins. A multiple choice test at the end of a Know Your Security online compliance training program is an example of a knowledge test to determine a person’s understanding

Evaluating Instructional and Noninstructional Interventions

255

of the role they play in protecting their organization from threats such as theft of intellectual property. Tests can include various types of questions such as multiple choice, true/false, matching, fill-in-the-blank, or written and can be paper-based or electronic. There is typically a “correct” answer against which learners are graded. A skill-oriented test attempts to measure the acquisition of, or proficiency in, a behavior. While a skill typically has a knowledge component(s) to it, a skill test focuses on demonstrating an observable skill. A driver’s test is not only a rite of passage for most teenagers, it’s an example of a skill test. Observation, discussed earlier, is often used with this form of evaluation so evaluators can watch for predetermined expected behaviors (braking, maneuvering, accelerating, hand position, parallel parking) and how well or poorly they are exhibited. Testing to obtain a driver’s license is a good example of combining tests—there’s a knowledge test to demonstrate an understanding of the “rules of the road” and in the vehicle skill test to demonstrate essential driving abilities. In organizations, testing is used in the learning process to certify individuals as capable of performing. Capability to perform and actual on-the-job performance are two things, which will be covered shortly. Organizations often use testing during and after learning interventions for employees who will enter new roles. Sometimes certification in a job role is required and even when not mandatory, the intent is to ensure a worker is fully prepared as they onboard and perform in their new role. In many organizations testing is used for compliance, sometimes required by government regulators. The validity of testing used and the maintenance of records are essential to prove compliance and avoid potential heavy penalties or worse. Organizations like Questionmark (www.questionmark .com) and Knowledge Advisors (www.knowledgeadvisors.com), through their Metrics That Matter platform, provide sophisticated technology-based assessment and testing databases, tools, and mobile applications. (See Exhibit 14.4 for a sample level 2 evaluation.) Level 3. While levels 1 and 2 primarily occur during or immediately following learning, level 3 in Kirkpatrick’s framework attempts to provide insight into applying learning on-the-job and occurs post-program and answers the question “did they apply it?” It helps to measure the transfer of learning that occurred between the learning event and some future point in time. The actual timeframe for conducting a level 3 evaluation can vary depending on several factors including the knowledge or skill involved, the opportunities afforded the learner to apply what they learned, the

256

Mastering the Instructional Design Process

EXHIBIT 14.4. LEVEL 2: EVALUATING LEARNING Directions: Please respond to the following statements by placing a check mark in the True or False column. Statement Managers should spend time to ensure that their employees have clear direction and understanding of their jobs. Managing people is time consuming. Therefore, managers should minimize the amount of time they spend with their employees. Employees respond well only to positive feedback. Building trusting relationships can help to minimize micromanagement. Employees need to have easy access to the tools and resources they need to do their jobs successfully. It is easy to be a good manager. People do not want guidance from their manager. They would prefer to figure it out on their own.

True

False

◽

◽

◽

◽

◽ ◽

◽ ◽

◽

◽

◽ ◽

◽ ◽

motivation to apply what was learned, and the level of interest and urgency key decision makers have in seeing the results of the level 3 evaluation. Level 3 evaluation can use several of the data collection methods reviewed earlier in this chapter including interviews, surveys, and observations. The learners themselves can self-report, through a survey, the extent to which they’ve applied the newly acquired knowledge or skill. Likewise, the direct manager, peers, and even customers who interact with learners can provide level 3 feedback on what they’ve seen the learner apply. Direct observation, by a training professional, manager, or peer, is another means by which to evaluate transfer of learning. Consider the short vignette below. As a new call center representative, Leslie went through three weeks of intensive training to learn both the service process of her new company, 800 Service Direct, and also “soft” skills such as dealing with an irate customer, active listening, and responding with empathy. After the training, Leslie fielded calls on her own. Her supervisor, Marco, used a common call center technology known as “double jacking” whereby he could listen in on her calls. After each call, Leslie and Marco reviewed and discussed what she did as well as how she

Evaluating Instructional and Noninstructional Interventions

257

could improve. Using an observation checklist, Marco filled in with notes during the phone calls. Over time, Leslie’s ability to handle calls, even the proverbial “curve balls,” increased to a point where she was fully proficient and able to handle calls entirely on her own.

This vignette illustrates a direct observation (using a standardized checklist) approach to evaluating the transfer of learning, but it goes a step beyond. It not only evaluates the degree to which Leslie applied what she learned during her three weeks of training, it also supported a coaching tool to engage an iterative process so Marco could assist her in getting from a lower level of competence to full proficiency. This is an example of a level 3 evaluation conducted relatively quickly after the initial training because there was a need to have Leslie fully performing in her new role quickly. The “costs” associated with incompetence were too great—even including the potential loss of valued customers. Further, this learning lends itself well to almost real time level 3 evaluation because the skills and knowledge in this case are an essential, if not complete, part of job performance. Other forms of learning, like a workshop on Creativity and Innovation, may lend themselves to level 3 evaluation that occurs many months following the learning event itself. Level 4. Also occurring postlearning, level 4 evaluation is the least frequently used level among the four, due to the time, cost, and resources typically involved and the expertise needed to do it effectively. Level 4 evaluation attempts to measure the impact of the learning (“Did it make a difference?”). This level goes beyond applying learning to determine the results or impact of that application. In the call center scenario described above, level 4 metrics might include things like customer retention, reduced call time, or increased sales. Level 4 is about organizational or business metrics that relate to quality, productivity, efficiency, sales, profitability, cycle time, turnover, performance, and other, typically quantifiable, measures. It’s difficult to conduct a level 4 evaluation when key metrics and baseline measurement are not identified and integrated up-front, during the needs assessment phase of the instructional design process. Designing and deploying a Building High Performance Teams program and then, after the fact, saying “Let’s do a level 4 evaluation on it” is a foolhardy endeavor for the professional instructional designer. Better to plan for, design, and set the stage for level 4 evaluation from the beginning of the project, by asking the customer questions about quantifiable results not being achieved or that are desired.

258

Mastering the Instructional Design Process

One of the issues that may arise when conducting level 4 evaluations is isolating the impact the learning had on the level 4 metrics compared to the impact that many other potential variables had (the economy, the job market, a new organizational strategy, change of leadership, new technology, or better tools and equipment). It is easy for others to poke holes at level 4 evaluation studies by pointing to a myriad of other factors, other than or besides learning, that could have influenced the results. The wise instructional designer will engage closely with others, such as business leaders, finance experts, and other subject experts when they embark on this evaluation endeavor because these individuals are well versed and actually “own” the organizational metrics under examination. Alternatives to Kirkpatrick Despite its widespread adoption in organizations, the Kirkpatrick model is not without its critics. Some have lamented that it was introduced nearly five decades ago and has changed little since that time. Others suggest that it is too linear or has an over emphasis on training when multiple interventions may be used and needed in a performance improvement situation. Some have gone beyond merely being critical of Kirkpatrick and have proposed enhancements or alternative approaches. Phillips’s (2011) ROI Model, Brinkerhoff’s (2005) Success Case Method. The Phillips ROI Model The Phillips (2011) ROI model extends beyond Kirkpatrick’s fourth level and adds a fifth level, which attempts to calculate the return-on-investment (ROI) of the intervention. ROI is a financial calculation that quantifies the financial value of the impact measures identified in level 4 (such as sales or productivity) relative to the costs of the intervention. The ROI formula is: ROI (%) = Net Program Benefits/the Program Costs × 100. Success Case Method Robert Brinkerhoff (2010) proposed an alternative to Kirkpatrick’s framework and called it the Success Case Method (SCM). This approach looks beyond traditional training interventions and also recognizes that many variables may be at play for performance and results. Brinkerhoff (2005) asserts that “Performance results can’t be achieved by training alone; therefore training should not be the object of evaluation” (87).

Evaluating Instructional and Noninstructional Interventions

259

The Success Case approach takes more of a holistic and systemic approach and suggests that the following questions be addressed: • How well is an organization using learning to improve performance? • What organizational processes/resources are in place to support performance improvement? • What needs to be improved? • What organizational barriers stand in the way of performance improvement? (Brinkerhoff, 88) Brinkerhoff’s approach to evaluation “combines the ancient craft of storytelling with more current evaluation approaches of naturalistic inquiry and case study” (91). Given its more qualitative nature, data collection methods associated with the SCM may include interviews, observations, document review, and surveys. As the name suggests, the Success Case Method attempts to identify individuals and/or groups who have applied their learning and who achieved positive organizational outcomes or results. Besides identifying successful examples, the SCM attempts to answer why they succeeded (what enabled success?). And while less glamorous, the method can also look at examples of unsuccessful applications or outcomes and then attempt to pinpoint the reasons for this lack of success (what were the barriers or challenges encountered?). Both dimensions can be useful in identifying stories that help to paint a picture and determine the worth or value of the intervention and how it can be improved going forward.

Reporting Evaluation Results An evaluation report can take a variety of formats and is used to capture the results so they can be communicated to various stakeholders. The following are questions that can help the instructional designer think through the best format to use for the report: • Is the evaluation report being requested by someone or is the designer initiating the report? In some situations the instructional designer is asked by someone to provide an evaluation report (a reactive situation). In other circumstances the designer assumes the responsibility for creating and sharing an evaluation report (a proactive approach). When an evaluation report is requested, the designer solicits input from the requestor regarding what is of highest interest and importance. This insight guides the focus of what is

260

Mastering the Instructional Design Process

included in the report. In proactive cases, the designer must try to anticipate what might be most valuable and useful to the recipient and this can help to guide the content and report creation. Where the designer is being proactive, there is probably nothing preventing a direct conversation with the potential stakeholder(s) to explain the intent and solicit information about what would be valuable to that person. • Who is the recipient of the evaluation report and what is their interest in the results? Return to the list of possible stakeholders and key questions these individuals may be interested in. It may be useful to separate primary from secondary stakeholders. Primary stakeholders are those who are the main recipient of the evaluation report and will make important decisions, like whether or not to continue funding. While difficult, an effective designer can “walk in the shoes” of the primary recipient and get well grounded in the information that person is most likely to be interested in, so it can be included in a way that is easy to interpret and act upon. While secondary stakeholders are important, the level of rigor and customization of the report is typically less than is afforded those who are primary. • What is my purpose in sharing this evaluation information? An evaluation report can serve many purposes and the wise designer will be intentional in his or her approach to achieve the maximum impact. One purpose may be to inform the stakeholder of the intervention itself. The evaluation report raises awareness of work that may otherwise not be visible or widely recognized. In this way, the evaluation report serves as a marketing or communication tool. Another purpose is to describe the impact of the intervention and justify the investment. Some stakeholders may ask “What happened as a result of that program?” or “What return did we get out of that initiative?” Armed with an evaluation report, a designer is equipped to answer those questions if asked or can even preempt the question by sharing the report proactively with key stakeholders. Another purpose is to stimulate an action to be taken. Action that might be taken could be internal to the learning organization such as making adjustments to the design or recruiting new facilitators. External actions could include things like expanding the deployment of a program to a different business unit or globally, decreasing funding, or deciding whether to outsource facilitation. • What preference does the stakeholder interested in the evaluation results have regarding how the information is presented? Evaluation results can be shared with stakeholders in a variety of ways. An evaluation report can be delivered with no opportunity for discussion or interaction with the stakeholder. Or, it may be sent as a preread and then discussed in an in-person or virtual

Evaluating Instructional and Noninstructional Interventions

261

meeting. Another approach is that the evaluation results can be presented to the stakeholder using a presentation program like PowerPoint or Prezi. Again, knowing the style, preferences, and expectations of the audience in advance helps to guide the manner in which the report is delivered. • What level of depth or detail does the stakeholder wish to receive? Similar to the manner in which the report is presented, another factor is the level of detail desired by the stakeholder. Just because a stakeholder is a senior executive does not automatically mean that the report should stay high level, as some senior leaders want to understand the specifics or may have questions about the details. An effective designer will seek to understand the needs of the stakeholder and then select what is presented in the report based on this insight. As revealed shortly, an executive summary is a way to pull together the key elements of the evaluation into a succinct snapshot that may be sufficient for some audiences while an appendix can contain data, details, or backup material.

Creating the Report This section describes a more traditional or formal written format that an evaluation report might take as a way to summarize the contents and sequencing that are typically found (Torres et al. 2005). This framework can be expanded or condensed and sections can be removed depending on the audience and intent of the report. Also, this framework can be used to create a stand-alone report or the contents can be converted into other presentation formats such as PowerPoint for the basis of verbally communicating select information to key stakeholders. Below are the main elements of an evaluation report. • Title page: Includes the report title, date, the author who conducted the evaluation, and the recipient of the report. • Executive summary: Includes the overall purpose and objectives of the evaluation, methodology used, and primary findings and recommendations or next steps. • Introduction/overview: Includes the background and context of the initiative, target audience, goals and objectives, methodology used (evaluation approach and data collection methods and sources), and any limitations of the evaluation. • Findings and results: Includes level 1 (participant reaction and satisfaction), level 2 (learning), level 3 (behavior change or transfer of

262

Mastering the Instructional Design Process

learning), level 4 (impact), and level 5 (return on investment) evaluation results or uses other organizing methods to objectively present the results (e.g., key questions, data collected, themes discovered, etc). Findings should balance both positive and suboptimal results. Metrics, visuals (charts, graphs, tables), and qualitative themes should be included. The report author should also include any barriers and enablers to achieving the objectives. • Recommendations and next steps: Includes the implications of the findings and results and summarizes the conclusions that can be logically drawn. Once recommendations are presented, any next steps/actions that can be taken because of these conclusions are shared. • Appendix: Additional details related to any of the above items, such as participant information, program materials, tests, surveys or other evaluation tools, and any other background information.

Disseminating the Report Once created, the next step is to distribute the report to stakeholders in the most effective way so the information in the report is most likely to be reviewed, processed, and acted upon. The timing of the dissemination is one consideration. If the report is distributed too long after the intervention, the value and relevance may be diminished. Likewise, if it is sent out at a time when it may compete with other priorities, such as the end or beginning of a business cycle, it may not get the attention it deserves from stakeholders. The method of dissemination, mentioned earlier, is ideally matched to the needs and desires of the key recipient(s). Sometimes using multiple methods addresses the varied needs of the audience and reinforces the key messages being conveyed. A short preread could be sent prior to an in-person meeting that incorporates a presentation and discussion, which is then followed by a full report. Media involves the vehicles used to distribute the report. A traditional approach is to use a word-processing software such as Microsoft Word to create a print-based report that can be printed or distributed electronically. Other software such as PowerPoint can create an evaluation report that incorporates graphics and animation. More sophisticated web-based tools can create a multimedia-based approach that lets the recipient engage with the material in an interactive and dynamic manner (rather than a onedirectional method).

CHAPTER FIFTEEN

REVISING INSTRUCTIONAL AND NONINSTRUCTIONAL SOLUTIONS BASED ON DATA

T

here are several reasons instructional designers engage in evaluation including providing key stakeholders with important information to raise awareness and to help them decide. Evaluation helps to ascertain learner satisfaction, knowledge/skill acquisition, transfer of learning leading to behavior change, and organizational results or business outcomes. Another primary objective of evaluation is to make revisions and improvements to the solutions based on data collected during the evaluation process. Strategies for making these revisions is the focus of this chapter. According to The Standards (Koszalka, Russ-Eft, and Reiser 2013, 60), “the competency itself is rated as essential since all instructional designers should be able to revise solutions based on data and feedback that are received.” The performance statements associated with this competency, one of which is advanced and two which are essential, indicate that instructional designers should be able to (Koszalka, Russ-Eft, and Reiser 2013, 60): “(a) Identify product and program revisions based on review of evaluation data (advanced); (b) Revise the delivery process based on evaluation data (essential); and (c) Revise products and programs based on evaluation data (essential).”

263

264

Mastering the Instructional Design Process

Types of Revisions Two primary categories of revisions are typically made. Product or program revisions involve changes to the content or messages being conveyed through the intervention. Besides the content itself, revisions could also be made to exercises, activities, assessments, sequencing or flow, and overall duration and length of segments. Adjustments to the delivery process include changes made to the manner in which the solution is deployed. Delivery processes could involve self-paced, job or performance aid, web-based, in-person, or “blended” approaches, which involve multiple methods. These types of revisions can encompass everything that happens before, during, and even after the intervention that pertains to how it is executed. Sources of Input for Potential Revisions Revisions to instructional and noninstructional interventions do not occur in a vacuum. There are many sources of input that can and should be sought and used. If a formal report of the evaluation results, discussed extensively in Chapter Fourteen, is created, this can serve as a source of input that the instructional design professional can use to identify changes. Direct feedback from participants, facilitators, training managers, logistics support professionals, and others is a way for designers to hear directly from stakeholders about what could be improved through making changes. Such direct stakeholder feedback can be solicited in real time as the intervention is being deployed, immediately after, or at a later date through a formal debrief discussion. Another source of input is change in professional practices that have occurred in the organization, which require updates to the program. A major upgrade to an organization’s performance management system may necessitate significant revisions to the training that supported the previous version. A final source of revisions is change to organizational policy. An organizational policy decision that no employee air travel can be booked within two weeks of flying or a decision that nonbusiness critical travel is restricted to only two trips per year are examples of sources of input that could lead to potential revisions, perhaps to delivery practices related to participant and facilitator travel or introducing virtual delivery methods.

Revising Instructional and Noninstructional Solutions Based on Data

265

Reviewing Evaluation Reports and Other Data to Identify Needed Revisions You may recall that Chapter Fourteen described the common elements in an evaluation report, which can take a variety of forms and can be delivered in a variety of ways. If an evaluation approach like Kirkpatrick’s four-levels framework is used, the report will include insights (and corresponding recommendations) related to participant satisfaction with various aspects of the intervention, the knowledge and skill acquisition, transferring learning and corresponding behavior change, and the business and organizational impact. Ideally, the evaluation report will suggest revisions to the content and provide program delivery suggestions. The participant reaction to the registration process may have been negative, suggesting that improvements are needed. An evaluation report may reveal low test scores on a particular item, which may necessitate the need for more descriptive content or examples in an online learning module. When such revisions are not called out directly in the evaluation report, the instructional designer may need to review the findings, recommendations, and the raw data to dig deeper and identify potential adjustments that may be helpful. Revising Delivery Processes to Improve Outcomes The delivery process is the way the intervention is implemented. Delivery processes span the time horizon of activities that occur before, during, and after the intervention. Delivery processes that happen before the intervention may include but are not limited to registration, communication (welcome, overview of the intervention, logistical details), prework (readings, assessments), instructions, program material, and so forth. These processes should make the rollout of the intervention more streamlined and should enhance the likelihood of achieving the instructional objectives. Assessing and Predicting Efficacy of Potential Revisions Once potential changes to the content or the delivery process have been identified, the instructional designer will be tempted to simply make these changes and move on. It is critical, however, to stop and strongly consider the likelihood of success of the adjustments. What are the potential implications that may stem from the change? How will participants and others respond to the change? Might there be any unintended consequences or

266

Mastering the Instructional Design Process

that must be addressed? Are the changes major or minor? What’s the best way to phase in the changes—all at once or more gradually? Perhaps most important, will the proposed change lead to the desired outcomes? These and other questions help to anticipate the efficacy of the potential revisions so that the designer can be more confident in them before they are made. If doubts arise when answering these questions, this can be stimulus for the designer to adjust plans so better outcomes are likely to be achieved through the potential revisions. Revising the Content of Products or Programs to Improve Outcomes Once proposed revisions have been finalized, decided upon, or approved if needed, it is time to implement the revisions to the content or programs to achieve better results. One type of revision is to the content, which includes the key messages, narrative, and substantive information in the product or program. Revisions to content may include shortening or lengthening the volume of the content to either reduce or lengthen the coverage provided so it is matched to what is needed most by the learners. Revisions may also include the means by which the content is made more clear, accurate, or relevant to the learners in order to better achieve desired outcomes. Improving clarity involves making the messages easier for learners to understand or comprehend. Improving accuracy includes correcting inaccurate information. Improving relevance deals with adding material or examples that make the material more meaningful to the learners and more closely linked to on-the-job performance. Besides changes to the content, another revision is to exercises, activities, or methods by which the content is delivered. When an intervention is piloted or deployed and feedback is gathered, there may be problems with some of the instructional methods used. First, changes may be made to the instructional method originally chosen. The original design of a Leveraging Inclusive Diversity program may have included a large group discussion of a topic. Through the evaluation, it may be determined that participants are reluctant to speak up in front of the full group. The program design can be revised so small group discussions followed by large group report-outs are used instead of going straight to the large group discussion. Second, changes may be made to the instructional method itself. Part of the design of a Developing Your Career workshop includes a card sort activity where participants select 10 cards (out of a possible 50) that best reflect what they enjoy doing most at work. Through the pilot, it is determined that participants are struggling with some of the items on the cards. Through this process, it’s determined that two major revisions must be made to the

Revising Instructional and Noninstructional Solutions Based on Data

267

activity—the descriptions on the individual cards must be clearer and there needs to be greater differentiation among each of the cards. As discussed in Chapter Twelve, most instructional programs have participant guides, instructor or facilitator guides, presentations, and other material that make up the complete instructional package. Another focal point for potential revisions involves changes that may be made to such material. These changes may be needed for various reasons. There may be lack of clarity in the instructions, an exercise, or the way content, material, or messages are presented. Any time learners, as well as facilitators, become confused about what to do, what something means, or how something is relevant or meaningful, then the instructional designer must take action to remedy this and bring greater clarity. Any time inaccuracies, with grammar, punctuation, wording, or spelling exist, they must be corrected to ensure material is credible and useable. Graphics or visuals that are used in participant guides, posters, or presentations may be unclear, inaccurate, or should not have been selected in the first place and must be corrected. By the time a complete instructional package can be ready for use, it must be error free, clear, accurate, and relevant. When this is the case, the designer can be confident that the solution is poised to succeed with a higher probability that the intended outcomes will be achieved. This confidence engenders a sense of pride and satisfaction in the instructional designer.

Gaining Stakeholder Support for Revisions The instructional design professional doesn’t work in a vacuum at any of the ID process stages. Many parties with varying, and sometimes conflicting, interests and priorities are involved in any design project. While it may seem insignificant compared to other key decisions made throughout the process, support for revisions must be garnered. Minor revisions, such as correcting spelling or formatting errors, may not require extensive stakeholder signoff and are often implemented by the designer and perhaps the manager. More substantive revisions such as increasing the length of a program, using externally procured content that requires additional funding, or adding a new exercise like a case study, may need additional stakeholder signoff. The first step in gaining support is to identify who is likely to have a vested interest or a say in the change. Once these individuals or groups are identified, a strategy for securing their support is developed and then executed. Stakeholders are fairly easy to identify, as they have likely been involved in the process already. However, sometimes a new stakeholder

268

Mastering the Instructional Design Process

is identified and must be engaged. Strategies to secure stakeholder support for revisions are usually fairly straightforward and involve engagement approaches ranging from written communication, to a group meeting, to a one-on-one formal or informal interaction. • • • Whatever the engagement approach used, the goal is to receive approval for the proposed changes. Some organizations require formal approval such as a written sign-off authorizing the changes, while others are more informal, in which verbal confirmation of the change is secured. In less formal environments, this often suffices, but it may be useful to document the approval as a way to memorialize this as a future reference point. See Exhibit 15.1 for a list of stakeholder questions to assist in improving the intervention. • • • EXHIBIT 15.1. QUESTIONS TO ASK STAKEHOLDERS TO IMPROVE THE INTERVENTION Objectives 1. Were the right objectives developed? 2. Were the right people involved in the development of the objectives? 3. Were any objectives missing? Design of the Intervention 1. 2. 3. 4.

What went well with the design of this intervention? Did this design support meeting the objectives? Was this the right design for the intended audience? Did this design have the right balance between time needed away from work and meeting development needs? Was the intervention too long? Too short? 5. What could have been better? Development of the Intervention 1. Were the correct materials (books, videos, handouts, job aids, etc.) used to develop this intervention? 2. Should there have been more/fewer materials used in this intervention? 3. Did the materials used in this intervention match the audience? 4. Did the materials enhance or detract from meeting the objectives? How? 5. What could have been better?

Revising Instructional and Noninstructional Solutions Based on Data

269

EXHIBIT 15.1. QUESTIONS TO ASK STAKEHOLDERS TO IMPROVE THE INTERVENTION, cont’d. Implementation 1. What went well with the implementation of this intervention? 2. Was the right pace used during implementation? 3. Were the right people involved in the implementation (facilitators, instructors, trainers, etc.)? 4. Could this intervention have been delivered in a better way? Explain. 5. What could have been better? Evaluation 1. 2. 3. 4.

What went well with the evaluation of this intervention? Were the right criteria evaluated? Were the evaluation strategies the best strategies? How could the evaluation have been better?

Overall Feedback 1. How well were the objectives of this intervention met? 2. What would you repeat or keep the same the next time this intervention is used? 3. What piece of advice would you give to improve the overall effectiveness of this intervention?

• • •

Implementing Revisions to Delivery of Products/Programs Once approval has been received, the revisions are implemented. The nature, complexity, and the number of changes being made determine the approximate time required. The change also determines the resources required for implementation. An editor or graphic artist may be the one to change the material or visual aids. A subject matter expert may be needed to develop or alter content. A procurement professional may be needed to purchase a new off-the-shelf tool or activity identified. The training administrator or Learning Management System (LMS) team may be needed to change the delivery or technology support. Often, revisions needed fall to the instructional designer to make. Regardless of the change and who is making it, the instructional design

270

Mastering the Instructional Design Process

professional typically oversees the changes and is ultimately accountable for ensuring the changes are made as intended. Evaluating Effectiveness of Revisions Once revisions have been implemented, the question becomes “did the changes lead to improved outcomes?” To answer this question, we can return to the four levels of evaluation and determine whether the revisions were effective. Were participants more satisfied with the modifications? (Level 1) Did test results improve? (Level 2) Was there a greater transfer of learning from the classroom to the workplace? (Level 3) Were business results and organizational outcomes achieved? (Level 4) The effectiveness of revisions can also be evaluated by many other means beyond the four levels framework. Making Revisions in a Rapid Prototyping Environment In today’s hyper-paced environment, long cycle times of development that require many months to design, test, adjust, and implement are too slow, and therefore often not acceptable. Instead, rapid prototyping is becoming more commonplace in the world of instructional design. In such environments, sometimes a less-than-perfect product is enough to begin piloting and gauging effectiveness. Revisions are made, almost in real time, and are followed by additional “pressure testing,” evaluation, and adjustments. In this way, cycle times may be greatly reduced compared to traditional design approaches where a great amount of time is spent up-front “getting it right” before the solution is piloted, evaluated, and adjusted. Whether due to client sense of urgency for faster speed to market, the emergence of technology and design tools that facilitate more rapid design work, or the new norm of instantaneous everything, it is more common for the instructional design professional to work at breakneck speed using rapid design methodologies. Striking the balance between rigor and speed is an ethical dilemma sometimes faced by practicing instructional designers. While it’s important to adapt and respond to faster and faster cycle times, the instructional designer must also maintain the course of the core ISD process that balances efficiency with effectiveness, which may get shortchanged, unless the designer holds firm.

CHAPTER SIXTEEN

IMPLEMENTING INSTRUCTIONAL AND NONINSTRUCTIONAL INTERVENTIONS

O

nce all of the initial design, planning, pilot testing, adjustments, and other preparations have been made, the intervention is ready to be implemented and disseminated to the organization and intended audience. Once a vision of change that aligns learning and performance goals with organizational goals is established, plans for deploying the intervention are created. From here, it is launch time and the implementation plans are executed and the intervention is disseminated. Especially early on, deployment efforts are carefully monitored under the watchful eye of the instructional designer and other key partners to ensure everything is on track and going smoothly. If and when adjustments to the deployment efforts are needed, they are identified and made accordingly so that desired goals are achieved efficiently and effectively. According to The Standards (Koszalka, Russ-Eft, and Reiser 2013), “This competency is considered to be advanced, and the performance statements are rated as being managerial or advanced” (61). The seven performance statements defined by Koszalka, Russ-Eft, and Reiser (2013, 61) are shown below:

1. Create a vision of change that aligns learning and performance goals with organizational goals. 2. Plan for the implementation of the interventions. 271

272

Mastering the Instructional Design Process

3. 4. 5. 6. 7.

Plan for the dissemination of the interventions. Plan for the diffusion of the interventions. Disseminate the interventions. Monitor implementation, dissemination, and diffusion progress. Identify required modifications to implementation, dissemination, and diffusion processes.

This chapter is all about execution—implementation of the instructional or noninstructional intervention to the target audience to achieve the intended change and desired performance or organizational outcomes. Implementation involves planning the logistical aspects of the deployment. The logistical plan includes “the personnel and resources needed, and it must indicate the needed time for learners. If planning for face-to-face sessions, the plan must also discuss the location and any needed arrangements” (Koszalka, Russ-Eft, and Reiser 2013, 61). The Standards suggests that the instructional designer not stop at implementation of the intervention and introduces the importance of planning for its dissemination, which is considered a managerial performance. Dissemination involves the means by which the intervention is spread throughout the client organization. Koszalka, Russ-Eft, and Reiser (2013) note that “the instructional designer must work with management concerning the timing and scheduling of the intervention. Depending on the organization’s fiscal year, certain periods may pose problems for the intended audience. Furthermore, coordination with other events and activities is critical” (61). According to The Standards, “Planning for implementation and dissemination are critical steps, but the savvy instructional designer also plans for diffusion of the intervention. Diffusion goes beyond dissemination and involves activities and processes to encourage widespread and long-term adoption and buy-in. Diffusion may include various instructional and communication strategies to support the proposed change” (Koszalka, Russ-Eft, and Reiser 2013, 62).

Creating a Vision for Change As Stephen R. Covey (1989) suggested for personal and leadership effectiveness in his book Seven Habits of Highly Effective People, instructional designers, too, must “Begin with the end in mind.” One of the most critical leadership competencies is the ability to create a clear and compelling

Implementing Instructional and Noninstructional Interventions

273

vision for the change they are trying to achieve in the organization. Effective instructional designers must exhibit effective leadership by doing likewise. A vision for change is a clear articulation of the future—a depiction of what will happen because of the intervention in terms of the end outcomes and results, and the behaviors, skills, knowledge, and attitudes of those in the target audience. The more vivid, clear, simple, and compelling the vision is, the more people will understand the desired future state and work to achieve it. The more a vision for change is confusing, complicated, or not relevant, the less likely the change will occur. See Exhibit 16.1. Aligning the Learning and Performance Goals with Organizational Goals Throughout the design process, instructional designers must constantly ask themselves how the learning and performance goals support the overall goals of the organization. As interventions are being envisioned, designers must also ask themselves how the intervention will support the learning and performance goals. It must be remedied if it is impossible to articulate how the intervention supports the learning and performance goals and how these goals support the overall goals of the organization. Misalignment can lead people to question why an intervention is even being undertaken. Clear alignment ensures relevance of interventions and should be considered throughout the design process by making connections back to these goals.

EXHIBIT 16.1. VISION FOR CHANGE CHECKLIST Question What is your “end state” goal? How will this goal help you to achieve your overall organizational goals? How will each stakeholder benefit? What will change as a result of this initiative? What are the expected outcomes? Who is going to be impacted by this initiative? Consider and list the potential “unintended” outcomes. How will this change the work of various stakeholders?

Answer

274

Mastering the Instructional Design Process

Benefits of Alignment and Challenges of Misalignment When learning and performance goals are tightly aligned with organizational goals, the greatest benefit is the dramatically increased likelihood of the intervention having an impact on the most critical measures of organizational success. The linkage starts with improved learning, which results in improved performance, which leads to better organizational outcomes. In this manner, not only will learners be more effective because of the intervention, but other stakeholders such as sponsors and executives will benefit because organizational goals (profitability, revenue, market share, customer retention, etc.) are likely to be attained. Much has been written about goal alignment in the literature on performance management. One form of alignment is vertical alignment, where goals at the highest organizational level are broken down into smaller component parts and “cascade” down through an organization. In this manner every person can see how their individual objectives, tasks, and work contribute to the goals of the organization. Figure 16.1 depicts this cascading process. As its name suggests, horizontal alignment entails ensuring that goals line up laterally across the organization. Horizontally aligned goals ensure that the objectives and work of different functional areas, business units, and teams all drive toward the same overarching outcomes. An often-used term in organizations today is the word silo, used to describe when an individual business unit, department, or team becomes insular and overly focused on its own goals to the exclusion of other areas. One author recalls a CEO who, jokingly, quipped that it was fascinating how all of

FIGURE 16.1. Vertical Alignment Division A Goal 1 Goal 2 Goal 3

Department X Goals support Division A’s Goals 1 & 2

Department Y Goals support Division A’s Goals 2 & 3

Department Z Goals support Division A’s Goals 2 & 3

Implementing Instructional and Noninstructional Interventions

275

her direct reports achieved their goals, yet the organization did not! “Funny how everyone can achieve their goals except me!” This insular way of thinking and operating becomes detrimental the more interdependent an organization is—and most modern organizations are more, rather than less, systemic with numerous interrelated processes, systems, and capabilities. Engaging Stakeholders and Negotiating to Achieve Alignment A common imperative stated by many HR, learning, and instructional design professionals is “you need to align to the business.” While it’s easy to agree with this and to recognize the benefits and value in doing so, many are left wondering how to do this. For starters, instructional design professionals must first identify the right stakeholders with whom to engage. For aligning learning and development goals with organizational goals, managers and senior executives are often important stakeholders. Other stakeholders may be helpful as well, such as the key client (which may be the senior executive), subject matter experts, or other learning managers or professionals. Once identified, the designer must be able to interact with these stakeholders, which is easier when there is already a relationship established. If it’s a new relationship, establishing trust and rapport quickly is critical. Some of the important interaction skills include listening, verbal and written communication, questioning, negotiation, and sometimes conflict resolution. A synonym for the word align is support because the goal of alignment is to ensure that the goals of the learning or instructional intervention support the goals of the organization. Learning goals should support on-the-job performance goals that ultimately contribute to attainment of organizational goals. The clearer, easier, and closer the “line of sight” by which the outcomes of the intervention can be seen in achieving the goals of the organization, the greater the alignment that exists. Misalignment is when the opposite is the case and stakeholders, such as participants, are left asking “Why are we doing this?” or executives begin saying “I don’t see the return we’re getting on this investment.” Once stakeholders are identified and the target of alignment is determined, the instructional designer engages in interactions to make the linkages between organizational goals and the goals of the intervention. This is where effective interaction skills come in to play. The designer may ask a senior executive client, “What results would be delivered if participants were performing at the highest level?” and “How would

276

Mastering the Instructional Design Process

this contribute to organizational outcomes?” Listening skills become important as the instructional designer attempts to understand what is meant by the stakeholders’ explanation. Sometimes the stakeholder’s response leads to follow-up questions to probe further and to better understand what was said. Restating what was heard, either verbally or as a follow-up confirmation in writing, helps to ensure understanding and validate alignment. What happens if the instructional designer perceives that alignment does not exist between the goals of the intervention and the goals of the organization? Sometimes this necessitates changes to the intervention to achieve closer linkage. In other cases, it may require negotiation skills so the instructional designer works to reach agreement with the stakeholder regarding reasonable expectations for the intervention. If an organizational goal is revenue growth of 20 percent, a learning intervention might be developed to equip sales professionals with skills to understand and cross-sell a new product. The instructional designer may need to negotiate with the stakeholder to agree upon the reasonable contribution the solution will achieve. In such a conversation, it may be pointed out that many variables, beyond sales people’s knowledge and selling skills, contribute to increased revenue. This may lead to additional noninstructional interventions that support the organization’s goals or it may help lead to agreement on the extent to which the instructional solution can contribute to the goal. Negotiation is a back-and-forth discussion process, underscoring the need for excellent communication skills and a good amount of patience and persistence to arrive at the outcome of aligned goals.

Planning for the Implementation and Dissemination of the Intervention Once the instructional designer is confident that goals are aligned and have received necessary stakeholder approvals, an implementation and dissemination plan guides the deployment of the solution. The specifics of the implementation plan vary depending on the intervention, but there are also essential elements that must be in place regardless. The most common interventions include self-paced, in-person, technology enabled, or blended types.

Implementing Instructional and Noninstructional Interventions

277

Self-Paced Interventions In self-paced interventions, learners are largely the key drivers of when and where they engage the interventions. In the early years of self-paced learning, paper-based instructional materials were most commonly used. This was followed by delivery of content through technology such as computer-based programs and video instruction. Today, most self-paced learning is conducted through the Internet or another web-based platforms. The common theme, though, is that the learner dictates when and where to engage, providing a great deal of flexibility and overcoming many barriers and reducing expenses, such as travel, associated with other forms of delivery. In-Person Interventions Face-to-face interventions involve people coming together in person in large or small groups to participate in the intervention. Learners may need to travel to a host destination to participate or the facilitator or instructor may travel to the learners’ location. Either way, such high touch experiences may drive up expenses to realize the benefits associated with in-person interaction. A variation of the in-person approach is the live virtual delivery where facilitators and learners are interacting together, but not face-to-face. In such cases, virtual technologies such as WebEx, GoToMeeting, and AdobeConnect are used to achieve the benefits of people working directly with one another. These approaches are more scalable and cost-effective, often without much, if any, degradation of the quality of the intervention or its ability to achieve similar outcomes as an in-person experience. Blended Learning Some implementations incorporate multiple delivery approaches, such as online and in-person. Blended learning approaches can achieve both the benefits of in-person delivery (networking and collaboration among participants, opportunity for practice, feedback, and coaching, and direct access to the facilitator for questions and answers) and the benefits of online learning (geographic reach, reduced in-person time, decreased costs, scale of deployment). Blended delivery is more complex with more requirements, resources, and moving pieces so ensuring smooth implementation can be more difficult than with other methods.

278

Mastering the Instructional Design Process

Aspects of Implementation When creating an implementation plan, there are many dimensions to consider. Many issues must be addressed “behind the scenes” and are less visible than the main event of an in-person workshop or facilitated learning experience. It’s only when one of these items is overlooked, goes awry, or otherwise has problems associated with it that it moves from being behind the curtain to center stage and, even if addressed, can leave a stain on the overall experience rendering it less positive than if it were avoided. One dimension of implementation is the people involved—both the number of people resources and the people themselves. Depending on the intervention, there can be more or fewer people involved and roles played. In large scale, multifaceted, or longer-term implementations, large numbers of people playing different roles come together to execute the intervention successfully. Deploying a Leading Change program over multiple geographies, using a blended learning approach and a train-the-trainer model for in-person delivery in a short time frame may involve many resources performing many unique and critical tasks. In smaller scale, simpler, or shorter-term interventions, fewer resources may be necessary. To execute a one-time delivery of an Interviewing Skills workshop for a group of eight new hiring managers using an internal talent acquisition subject expert, fewer resources are needed. Many roles may be required when implementing an intervention. Sometimes, such as in smaller or budget constrained organizations, individuals involved may be tasked with multiple roles, whereas in larger organizations, individuals or teams of people may specialize in various areas important to delivery. Sometimes roles may be outsourced and performed by external resources. Similarly, resources may reside in the organization’s learning function or they may be in virtually any area of the organization. These and other factors determine both complexity and the capabilities and expertise available to execute the intervention. Table 16.1 displays a listing of some of the many roles that can be played by various individuals during the implementation phase. The instructional designer may play the role of, or at least play a part in, identifying which roles are needed, determining who is best suited and available to perform those roles, and enlisting them to do so. Some implementation efforts may use existing and well-established processes such as the process used to print and ship materials to a location. Other projects or subelements of a project may be new or unique to that implementation. A new initiative may involve requesting and selecting

Implementing Instructional and Noninstructional Interventions

279

TABLE 16.1. PEOPLE-RESOURCES NEEDED DURING THE INSTRUCTIONAL DESIGN PROCESS Before the Intervention

During the Intervention

After the Intervention

Subject Matter Experts

Provides information regarding content

Helps to course correct if needed

Communication Specialists

Creates communication plan to identified stakeholders Creates timeline and rollout plan for intervention Collaborates with various stakeholders to bring resources together Completes needs assessment Creates intervention objectives Designs and develops curriculum Creates evaluation plan Pilots the intervention Communications with various stakeholders Supports the instructional designer (design of exercises, predicts audience reactions, design room layout, technology needs, etc.) Provides graphic design support during the design and development of the intervention

Supports evaluation and needed edits to curriculum Supports learner application and follow up Supports communication components of the evaluation plan Provides project management support for the evaluation plan

Aids in any course correction throughout the intervention

Provides participant support as they transition back to the workplace Conducts evaluation

Project Managers

Instructional Designers

Facilitators

Graphic Designers

Reports evaluation results

Facilitates the intervention Provides one-on-one coaching

Supports the instructional designer with the evaluation plan

280

Mastering the Instructional Design Process

TABLE 16.1. PEOPLE-RESOURCES NEEDED DURING THE INSTRUCTIONAL DESIGN PROCESS, cont’d. Before the Intervention

During the Intervention

After the Intervention

Technology Experts

Provides technology design and set up

Provides technology support

Business Leaders

Provides business perspective during intervention design Provides funding and resources Sponsors or markets the intervention Removes barriers to success

Provides support (guest speaking, coaching/mentoring participants) Intervenes with unexpected issues

Removes any technology used during intervention Reviews evaluation results Decides next steps

volunteers to play the role of facilitator rather than the typical approach of using professional facilitators. Such a situation may necessitate a new process being established by which to make this happen. Especially with new, modified, or unique implementations, but with well-established processes for roles to be performed flawlessly, several factors must be covered to ensure smooth implementation. A great deal of time and attention must be spent in up-front planning, contingency planning, process mapping, role clarification, hand-off and transition management, problem identification, and escalation and attention to dependencies and integration of efforts. Train-the-Trainer Approach to Dissemination An approach sometimes used for implementing an intervention is the train-the-trainer method. This typically involves a master trainer who “certifies” or prepares a less experienced training or nontraining professional to deliver content or instruction. The train-the-trainer approach enables a greater number of people to be involved in the direct dissemination of the intervention. Once prepared, the certified individual may be involved with delivery on a full-time, part-time, or periodic basis depending on needs, capabilities, and demands. The master trainer may be internal to the organization, such as an internal subject expert, or an external consultant or consulting firm, which holds intellectual property rights to a program or content.

Implementing Instructional and Noninstructional Interventions

281

An early step in a deployment strategy that uses train-the-trainer is to determine the number of certified trainers required to deploy the initiative over time. A key consideration in this calculation is to examine the implementation plans and specifically the number in the target audience, the average size of the session, and the anticipated time frame. To illustrate, an organization may attempt to shift from a culture of advancement based on waiting for the “tap on the shoulder” to one of career self-management. To support employees in this shift a variety of tools, process changes, and supports are identified including a one-day Navigating Your Career workshop. The organization has 10,000 employees and feels strongly that everyone, regardless of level or tenure in the company, can benefit from attending and makes it required training. The instructional design team, working with managers, employees, and talent acquisition experts, creates and pilots a program implemented and disseminated to the organization. It is decided that a train-the-trainer approach will certify human resource and recruiting professionals to deliver the sessions and also to serve as on-the-ground resources following the rollout. Key decision makers, considering various factors such as business cycles, resource availability, time of year, and other priorities land on a two-year implementation time frame. An important consideration is the average class size, which is determined to be 25. Based on these factors and assumptions, what follows is one means by which to calculate what resources will be needed. If 10,000 employees will attend in groups of 25 participants on average, there will need to be roughly 400 individual sessions (10,000/25 = 400 sessions). Given the two-year time frame, this means 200 sessions will be conducted each year. Looking at the trainers’ other priorities and what is reasonable in terms of capacity, it’s determined that delivering a one-day session twice per month is reasonable. This equates to 10 percent of the person’s time in actual delivery (eight hours of delivery every two week period of 80 hours: 8/80 = 10 percent). To get a realistic picture of total time, delivery, and nondelivery, the number should be increased to 15–20 percent total, depending on the amount of preparation needed, outside of session involvement (such as classroom setup), and level of post-session support for participants. While an annualized capacity of 24 sessions (2 per month × 12 months) per trainer per year could be calculated, a decision is made to deflate this number to 20 to account for vacations, holidays, and other unforeseen activities that would compete with classroom time. With 20 sessions able to be reasonably delivered by each trainer, 10 trainers must be certified and dedicated to delivery over two years (200 sessions per year/20 sessions per trainer = 10 trainers needed).

282

Mastering the Instructional Design Process

The previous example shown is fairly straightforward; there are other factors and dynamics that can complicate matters and must be accounted for when applicable. The geographic dispersion and the trainers is one factor. If trainers are in corporate headquarters, but 75 percent of the employees are elsewhere, significant time and expense may be needed for travel. This could also raise issues of suitable facilities and on-the-ground resources to support logistics and other delivery related activities. These considerations should be thought through when deciding the ideal delivery method, perhaps leveraging technology that is more efficient yet no less effective. Budget constraints are another factor. Ramp-up time for the facilitators and the ability of employees to be freed up to attend the training are others still. Another is the ability to schedule and deliver consistently rather than haphazardly. Monitoring the Quality of Implementation, Dissemination, and Learner Progress As interventions are deployed, they must be monitored for quality and to ensure that objectives are being achieved and delivery is smooth and efficient. Through monitoring, problems and issues can be surfaced and addressed as needed. More proactive approaches can anticipate and prevent problems before they become larger. An example of this would be instructor travel based on the delivery schedule. If the delivery schedule cannot be fixed for some period in advance, instructor travel cannot be booked and either instructor availability may become a problem or travel expenses may become prohibitive. This avoidable situation could be addressed by locking in the delivery schedule well in advance. Monitoring learner progress throughout delivery is critical. Kirkpatrick’s four-levels framework can be part of the monitoring system. If learners are dissatisfied (level 1) with some aspect of the program (food, logistics, registration, facilitation, etc.), decisive corrective action can be taken to remedy the situation quickly. If learners are not acquiring new knowledge and skills as evidenced by various forms of testing and assessment (level 2), then root-cause problem solving can figure out why this is the case. Perhaps the content must be changed, instructions must be clarified, examples must be incorporated, or exercises must be improved. If learners are found to not be implementing what they’ve learned in the classroom back in their jobs (level 3) there may be barriers being encountered in the work environment that must be removed or support systems must be in place to enhance transfer of learning. Finally,

Implementing Instructional and Noninstructional Interventions

283

if the expected impact (level 4) is not being achieved, then the reason(s) why must be uncovered and action taken so desired results and outcomes are realized. Monitoring the implementation and dissemination of the intervention can be thought of on two levels. Macro-level monitoring looks across multiple events and attempts to detect trends occurring. Comparing the evaluation scores related to instructor effectiveness over time can be a way to determine which instructors are more effective than others. Additional data about participants could be introduced and analyzed to determine which instructors may be better matched to particular types of learners. Monitoring may reveal that one instructor is more effective with front-line employees whereas another may be better suited for senior executives. The larger the scale and scope of the implementation, the more potential data can be collected and analyzed to surface trends, themes, and information, which tells the instructional designer a great deal about strengths and potential areas of improvement. Micro-level monitoring is the second way in which interventions can be observed. Such monitoring zeroes in on a variable rather than looking across many. Examining a newly certified trainer’s evaluation scores following the first solo delivery experience is one such example. Looking at test scores for the first delivery of a new program can reveal quick insights about the effectiveness of the content, instruction, or facilitator. An instructional designer sitting in on a delivery allows him or her to make direct observations about what is working or not working with an eye toward enhancements to various aspects of the design. Many people can be involved in various forms of monitoring throughout an implementation. Monitoring may be formal, such as a training manager running a level 1 evaluation report to track progress. It may also be informal such as an administrative assistant realizing that an office recently moved from one location to another necessitating a change of address in the system prior to materials being shipped. Regular review meetings to discuss key aspects of implementation and dissemination with report outs by key team members can be an effective way to both monitor progress and also connect the dots and achieve greater integration of efforts. The instructional designer may play the role of bringing the team together or it may be done by someone else. Frequency and duration of the meeting can vary depending on what needs to be covered, who needs to be involved, and where the implementation is in its life cycle. Brief and informal daily “huddle” meetings can be effective, as can more formal and longer review meetings held on a less frequent basis. When

284

Mastering the Instructional Design Process

problems surface, other communication vehicles may need to be activated, such as e-mail, text, videoconference, emergency meetings or other means by which the right people can be engaged and called into action. Learning Management Systems to Monitor Learner Progress and Course Completion A learning management system, or LMS, is a powerful internally or externally hosted software platform that forms the backbone of a learning organization. Depending on functionality, an LMS is used to create learning paths, prescribe required learning, register participants, track completions, launch e-learning courseware, manage participant and facilitator scheduling, and perform analytics and reporting. A primary capability of most LMSs is the ability to push learning out or prescribe learning to the target audience, and to enable learners to search a catalog of learning assets and register on their own. Mandatory or required learning, such as compliance or job-related training, is often assigned to the target audience through the LMS so it shows up in the individual’s learning plan as required until it is completed. Likewise, when a learner searches for a learning asset available via the LMS and self-registers for it, it also shows up on the learning plan until completed. LMS administrators, working with clients and leveraging organizational and employee data, can zero in on audiences or segments of the full population to target particular learning to individuals or groups, or automatic triggers may be set up. For instance, when someone is promoted or hired into a first-level supervisory role, the LMS may be set up to flag this individual and automatically assign a Management 101 type of offering. Sometimes there is a predetermined time frame by which learning must be completed. An easy example is completion of an annual Ethical Awareness and Decision Making online course assigned early in the year that must be completed by December 31. In other cases, no time frame is given for completion and in other situations, learners themselves, or their managers, set the target time frame based on the development plan. In any of these scenarios, a major benefit of an LMS is the ability to run reports and analyze data to determine learner progress and course completion. This information is used by the learning team and can also be included in reports and various communications with stakeholders and key decision makers. Using the annual compliance example, in the last weeks and months leading up to the end-of-year due date reports can show the percentage of employees who have not completed the learning, where they are located, and other attributes such as organizational level

Implementing Instructional and Noninstructional Interventions

285

or job function. These insights can be used to target additional communication or to engage in individualized conversations with employees themselves, their managers, or others, such as compliance or human resource professionals. Sometimes an LMS has measurement capability built in and in other cases, it may be “bolted on” so learning measurement and evaluation data can be captured, integrated, and reported upon using the system. In either case, having the ability to monitor and track evaluation results becomes a useful tool for the implementation team.

Planning for Diffusion of the Intervention Once initial implementation of the intervention has been achieved, there may be a need or desire to focus on full diffusion so greater sustainability and impact is achieved. Diffusion goes beyond implementation and dissemination and focuses on embedding the change even more deeply in the organization. When an instructional intervention is a key component or part of a more systemic change effort, diffusion is enhanced by identifying other supporting interventions that help enable the change to stick. An organization may implement a solution that focuses on building Coaching Skills for Managers to create a culture of feedback and coaching. Besides rolling out an instructional solution to managers, a diffusion strategy may involve several supporting efforts to achieve the objective of culture change. Managers identified as being exemplary coaches may be designated as point people who provide ongoing support to their peers as they attempt to build their skills. Coaching tools and job aids that managers can use before, during, and after conversations they are having represent another strategy that can further drive the change more deeply into the organization. Focusing on full diffusion requires additional planning, resources, and effort, but that cost must be weighed against the potential impact and upside of this investment.

Encouraging and Achieving Adoption and Buy-In A foundational concept in change management, and a primary goal of any intervention, is adoption among the target audience. Adoption is the degree to which the change is accepted and implemented by the target of the change. For example, an intervention includes the deployment of a job aid or tool to assist a customer service representative in upselling a warranty to

286

Mastering the Instructional Design Process

customers calling in, generating revenue for the organization. To achieve these objectives, the rep must use the tools to gain proficiency. If adoption is not achieved, then fewer sales are made and organizational goals such as revenue or profitability suffer. Several dimensions are associated with adoption of any new change. How quickly the intervention is adopted, or the speed of adoption, is one area. Another is the depth of adoption, which involves how fully or completely the change is embraced. Finally, the quality of adoption deals with the level of proficiency achieved among the target of the change effort. Several factors can accelerate or impede adoption and buy-in. The more relevant the intervention is to the intended user, the higher the acceptance and usage is likely to be. The clarity provided helps the user understand what to do and how to do it, therefore increasing adoption. The more support, from managers, peers, or others, that is available or provided, the greater the buy-in and implementation. When distractions that interfere with the performer’s focus and ability to apply the change are removed, adoption is higher.

Compliance versus Gaining Commitment When attempting to gain adoption and buy-in as part of a change management effort, two opposing approaches can be taken. One is to drive toward compliance by requiring or forcing people through pressure, coercion, or inflicting some sort of penalty to push someone to change. While compliance may occur, the level of commitment or ownership among the audience or recipients of the change will likely be low. Little buy-in to the change exists. A compliance-oriented approach to the change may be more expedient, but the effectiveness and staying power of the change is questionable. Another approach is to take steps that will lead to greater commitment to, or ownership of, the change. Several strategies can garner higher levels of commitment. Involvement in the process is a proven strategy that drives greater degrees of ownership. When stakeholders are asked for input, invited to play a role, or asked for their feedback, and other approaches are taken to engage them, the reciprocating response is higher commitment. Further, when those most directly in the process are involved in the design and deployment of the solution, the quality of the solution is likely to be higher because the expertise and wisdom of those closest to the situation have been included. An approach that engenders greater levels of buy-in and commitment may be less efficient because

Implementing Instructional and Noninstructional Interventions

287

gaining stakeholder involvement can be time consuming. Therefore, it’s important to think carefully about situations that would benefit more from having greater levels of ownership.

Monitoring Implementation, Dissemination, and Diffusion to Identify Potential Adjustments Implementation, dissemination, and diffusion efforts must be monitored to know if things are on track, if adjustments are needed and, ultimately, if success is achieved and objectives are met. Monitoring lets the instructional designer and other stakeholders know how things are going. The ability to monitor progress is made easier when objectives and outcomes of the initiative, established early in the design process, are well crafted, clear, and easily measured. When this is the case, it’s easier to determine whether progress is being made than when the objectives are nonexistent, confusing, or ambiguous. Monitoring creates a feedback mechanism to detect problems that must be corrected and also highlights what is working well so insights can be broadened to accelerate implementation efforts. It should be expected that any project will encounter obstacles, challenges, and problems. To expect flawless implementation, especially with large scale and complex initiatives, is foolhardy and not realistic. Monitoring can help to surface these issues so they can be addressed early and expediently and so progress toward the end goals is not hampered. There are various ways to monitor implementation efforts. It’s all about feedback. Chapter Fifteen introduces several data collection methods such as observation, interviews, surveys, and other approaches. These methods can also be used for monitoring. Besides these more formal methods, a simple yet powerful and frequently used approach is to listen, observe, and ask for feedback from all who are involved in the implementation efforts and who would share their perspectives. This “wide net” approach can gather real time insights in an informal but highly effective way. Another aspect related to monitoring is communication. Who needs to know the information and insights accumulated during the monitoring process? The instructional designer should determine who (who needs to know?), what (what do they need to know?), when (when do they need to know?), and how (how should they be informed?). The instructional designer should consider the stakeholders discussed previously, and the information believed to be useful to them in both judging the effectiveness of the implementation and helping them plan and decide, including making adjustments.

288

Mastering the Instructional Design Process

Taking Action on Deviations or Problems Surfaced through Monitoring Once potential adjustments are identified, the instructional designer makes a determination and, if warranted, plans for implementing identified changes based on monitoring efforts. Changes can be made to the learning or nonlearning intervention itself. If it’s determined that to achieve more full adoption of a new skill or behavior, a job aid with process steps articulated would be useful to deploy to the performers mobile devices, then this support tool should be developed and deployed. In other cases, changes may need to be made to the delivery process. Through monitoring efforts, it may be determined that individuals who went through a train-the-trainer process and were positioned to facilitate sessions are being pulled by managers to work on other high priority tasks and are no longer available to facilitate the sessions. In such a case, the delivery approach may be altered to one that deploys full-time training professionals instead. An alternative to this dramatic change could be for the designer to work with the managers of the trainers to revisit or renegotiate the expectations, to get them back on track or to make refinements that strike the right balance given the goals and constraints that are present. Whatever the change, once it’s decided that the benefits of the change are greater than the costs associated with making them, action is taken to implement the alteration. The actual adjustment may be made directly by the instructional designer or, more typically, it is made by others involved in the process such as the delivery team, the training manager, the instructional design manager, technology specialist, client, or several other potential individuals. The person closest to the change being made is likely the one who receives word of the change and then implements it. Similar to any organizational change being proposed or implemented, unintended consequences should be considered before making the change (Rothwell 2015; Rothwell, Stavros, and Sullivan in press). An unintended consequence is an action, reaction, or result that occurs and that was not anticipated. Unintended consequences can be positive, but they can also be negative and even create more or bigger problems than originally needed to be addressed. Avoiding such situations is sometimes as easy as being aware of the possibility of unplanned outcomes and asking where, when, and how they might occur. When a possible unintended consequence surfaces, the instructional designer determines whether it is important enough to address and, if so, the best means by which to do so. It’s a question of risk mitigation.

PART FIVE

MANAGING INSTRUCTIONAL DESIGN

Part VI: Predicting the Future of Instructional Design

Part I: Overview

Part V: Managing Instructional Design

Part II: Planning and Analyzing for Instructional Design

Part IV: Evaluating and Implementing Instructional Design

Part III: Designing and Developing for Instructional Design

CHAPTER SEVENTEEN

APPLYING BUSINESS SKILLS TO MANAGING THE INSTRUCTIONAL DESIGN FUNCTION

T

his chapter discusses the importance of applying business skills to the instructional design function. According to The Standards, “professionals who are proficient in this competency understand how instructional design fits within the larger organization structure, and they operate within that structure to produce quality results. Work in organizations is assigned to various functions based on employee expertise and the work that the organization needs performed. Often, the assignment of work is both political and competitive. Professionals who succeed over time recognize the importance of working in a function with other professionals who: • Contribute to the organization’s purpose defined by its mission, strategic plans, objectives, goals, and tactics • Establish and evaluate projects based on standards of excellence • Promote the function’s contribution to the organization using business cases that demonstrate value • Maintain staff including people who are both internal and external to the organization, who can perform the function’s work • Manage project finances effectively • Obtain and manage project staff by recruiting professionals with the required competencies, retaining those professionals, and developing those professionals to carry out the function’s work 291

292

Mastering the Instructional Design Process

• Promote the function’s services and manage relationships with internal and external customers and stakeholders. Instructional designers who demonstrate this competency work with their organization’s culture, structure, politics, and processes rather than against them. Those able to demonstrate these competencies are in a better position to influence the strategic thinking and actions of an organization, because they can help envision how organizations align the capabilities of their employees in response to a new strategic organizational objective. People are the most expensive resources of an organization and often the most challenging to redirect when an organization wants to change its course or increase its pace. Expert designers who are masters of these competencies can also become architects of change. Not surprisingly, by supporting systemic efforts they find it easier to be more successful than those who flail against the system” (Koszalka, Russ-Eft, and Reiser 2013, 64–65). With this background in mind, it is clear to see that applying business skills to managing the instructional design function is a managerial competency. It comprises seven performance statements, all of which are managerial: “(1) align instructional design efforts with organization’s strategic plans and tactics; (2) establish standards of excellence for the instructional design function; (3) develop a business case to promote the critical role of the instructional design function; (4) recruit, retain and develop instructional design personnel; (5) develop financial plans and controls for the instructional design function; (6) obtain and maintain management and stakeholder support for the design function; (7) market instructional design services and manage customer relations” (Koszalka, Russ-Eft, and Reiser 2013, 64–65).

Where Instructional Design Fits into the Larger Organization In larger organizations there may be an instructional design function where professionals reside and do their work as part of a team. Where this function sits in the larger organizational structure may vary. This section will cover common scenarios. At the highest level, instructional design functions may be part of the human resources department as part of a Corporate Center of Excellence or Expertise. Or, it may exist outside of HR and be embedded or part of the business. In still other organizations, it may be positioned in other places.

Applying Business Skills to Managing the Instructional Design Function

293

Human Resources Center of Expertise Model When instructional design is part of Human Resources (HR), it is often housed within organizational or talent development functions. Many organizations have adopted an HR model that breaks enterprise functions such as recruiting, compensation and benefits, inclusive diversity, and learning and development related groups into Centers of Expertise (COE), which are also sometimes called Centers of Excellence. COEs typically focus on setting strategy, creating processes, and designing interventions deployed through Human Resource Business Partners (HR BPs) who are aligned with each area of the business. The names of learning and talent Centers of Expertise vary and may be called enterprise learning, training and development, talent management, leadership development, organization effectiveness, or development, among many others. Often, instructional designers reside within such departments and employ their trade on behalf of that business or support internal department clients. Typically, the work of HR instructional designers spans the organization because they across organizational boundaries and serve the enterprise. Instructional design that supports the creation of an enterprise high-potential leadership development program is an example. This is not an exclusive focus of HR instructional designers who may also get involved in local projects. Embedded Model In many organizations, instructional design is embedded within a business unit. Unlike an HR Center of Expertise, which primarily focuses on enterprise initiatives or projects, business unit instructional designers typically report to non-HR areas. The focus is typically on design projects that are unique to one organizational part. Business-centric instructional design functions are often part of a functional or technical training department. Where such functions report can vary. In a manufacturing company, for example, the function could report to operations, quality, sales, supply chain, finance, safety, engineering, technology, or general management. As with human resources, in business unit training departments, instructional designers perform their responsibilities for that part of the business. Instructional Design Community of Practice In some organizations, there may exist part-time instructional designers who do their work only on a periodic basis. These “virtual” practitioners do not belong to an instructional design department. In such situations,

294

Mastering the Instructional Design Process

the phrase “strength in numbers” may be appropriate because the capabilities, capacities, and talents that exist among these individuals might be harnessed to unleash greater value than having them work in isolation. This harnessing process may be initiated by one or a few practitioners who spot an opportunity and lead the way. This person or group may reside in the corporate COE, an embedded learning function, or somewhere else. Often, this leadership results in creating a formalized or informal Community of Practice (CoP). The purpose, focus, and scope and the work, meeting frequency, format, and work output can vary widely, but it is often the result of the gap that exists, the opportunity that may be realized, and the interest and energy of the individual or group leading the charge. The University of British Columbia has an Instructional Design community of practice whose website (http://ctlt.ubc.ca/programs/ communities-of-practice/instructional-design/) attempts to attract prospective members by asking, “Are you an Instructional Designer who is involved in supporting faculty to improve their learning environment or teaching practice? Interested in knocking down walls and connecting with other Instructional Designers?” The site defines the CoP as “The Instructional Design (ID) Community of Practice aims to be ‘a self-sustaining community where like-minded members connect with others, online and face-to-face, about integrating technology, discussing hot topics, exchanging best practices, sharing the latest tricks for technology tools, trading examples of delivery methods, collaborating for research, exploring evaluation techniques, and much more.” The description of the community of practice at University of British Columbia depicts what a CoP in many organizations might represent. Its purpose is to create connections, share best practices, provide support and assistance to one another, and find opportunities to collaborate and add value that otherwise might not be created if they were working independently. of other practitioners. The value and benefits that any instructional design community of practice may realize can also vary widely. It may stop at creating social connections whereby practitioners learn other practitioners exist. Such superficial relationships may be deepened through structured or unstructured interactions, which may be accomplished in-person or virtually. These relationships may be strengthened further by more deliberate and purposeful interactions. Meetings and interactions amongst practitioners are planned, intentional, and focused on achieving objectives, outcomes, and benefits. The greater the frequency and depth of interaction among practitioners,

Applying Business Skills to Managing the Instructional Design Function

295

the greater the potential benefit to both fellow practitioners and to the “host” organization.

Contributing to the Organizational Purpose and Strategy An essential set of business skills that instructional designers must master is the ability to grasp the overarching purpose of the organization. Why does the organization exist, and who are its customers, markets, products, and services? What is the core mission, the short- and long-range strategies, what are it’s objectives, and the initiatives and tactical plans that will contribute to achieving those objectives in pursuit of its purpose? Understanding the culture, core values, and norms of behavior is also important because they help to define “how” work gets done and goals get achieved. The more instructional designers are steeped in the organizational priorities and business operations, the greater the chance they will contribute greater value. When instructional designers lack understanding it’s more likely that their work may become disconnected from business realities. One of the worst situations any learning or performance professional can be in is to be disconnected from the organization’s “mainstream.” This can lead to a vicious downward spiral where their work is not valued. It leads to lack of requests for support or real challenges in marketing of services, which further erodes credibility. If that persists, the organizational decision makers may decide that the function is not worth the investment and may outsource or eliminate it.

Establishing Standards of Excellence and Governance Some instructional designers are commissioned or take it upon themselves to establish standards of excellence and governance structures and processes for instructional design practices. Doing this creates agreed-upon standards by which instructional design processes, services, and practices will be carried out within the organization. By doing so, greater levels of efficiency and effectiveness can be achieved and variability and redundancy reduced. It provides a blueprint or roadmap for instructional designers to follow when carrying out their work. It also gives organizational stakeholders a clear set of expectations regarding instructional design support for their project. Standards help to raise stakeholder understanding. This

296

Mastering the Instructional Design Process

deeper awareness and education can create greater appreciation of instructional design and the rigor that must be exhibited to ensure that successful outcomes are achieved. Clear standards may also include the expectations instructional designers have of the stakeholders, increasing enrollment and even active engagement in the process, which is another key to success. There are other benefits to establishing instructional design standards of excellence in an organization. Where many practitioners are doing independent or isolated work across an organization, having standards can help to create a common language and expectations. It can also help to reduce variability and even raise the bar on quality and performance related to instructional design work. Greater efficiencies can be achieved because less “reinventing the wheel” may be required to achieve tasks. From a talent mobility standpoint, having shared standards of excellence can foster easier movement and transition of instructional design talent from one part of an organization to another because there is a common baseline of practice, which makes transitions smoother. This can foster career development opportunities among instructional designers because they may be more qualified for openings in other parts of the organization using the same standards they are operating under, which may improve one’s competitiveness for a new role. The words “governance” amd “standardization” sometimes cause a negative reaction among some stakeholders. Rightly or wrongly, governance can be associated with bureaucracy, inefficiency, slow decision making, and inflexibility. Unfortunately for those attempting to implement governance structures, these perceptions may be well grounded. Governance and standards can be mandated in a top-down manner or they may be implemented in a more grassroots, consensus-based manner by those involved with the processes being standardized. Knowing that some may be highly leery of such attempts is helpful because it can ensure that a filter is used so efforts to implement standards don’t become extreme and sacrifice the benefits. When attempting to implement standards, involvement by those closest to the process under exploration is suggested. Perhaps an organization is attempting to establish standards related to learning evaluation practices in an environment where various learning practitioners conduct evaluations in different ways. This decentralized and inconsistent approach creates inefficiencies across the organization. One way to achieve standardization in such a situation is having the organization’s executives mandate that a common evaluation tool, with standardized

Applying Business Skills to Managing the Instructional Design Function

297

questions and a centralized evaluation system or database, be used. An alternative approach to creating a standardized approach to evaluation, absent a mandate, would be to bring the practitioners together to agree on what should be common and standardized. Exhibit 17.1 contains a list of possible guiding principles that can be used when attempting to set standards or governance in instructional design practices. Once a practice is identified, whether by mandate or through consensus, as a candidate for becoming a standard, additional work must be done. A standard may be seen as one best way of doing things, but that may not always be the case. For various reasons, such as cost, complexity, resources, or time, a standard may not represent the best way of doing things, but rather will be adopted as the common way of doing things. Once it is decided upon, various parties involved must adopt that common practice, process, tool, or way of operating. Documenting the process or practice as it’s intended to be operated becomes an important way to not only capture the agreed-upon way of doing things, but also for communicating that to various stakeholders involved. When standards or governance is set in an organization, an accompanying plan of controls typically comes with it. As the name suggests, a control is a means by which to ensure the practice is implemented as designed, designated, or agreed upon. Controls may be self-administered, and even built in to the practice itself, so the users of the Standards themselves ensure consistency and application. An employee who operates a process in a pharmaceutical manufacturing process may have a checklist to ensure standard procedures are followed. Other controls may be measured and monitored by managers. In the pharma example, the manager of the process may observe whether the procedure is being followed and take corrective action if it is not. In other situations, an independent third party such as a quality control department or internal or external auditor may be responsible for objectively ascertaining whether the standard is being adhered to. When instructional design standards are established, they are typically governed or audited by the instructional designers themselves, their managers, or other members of the learning and development organization. Sometimes, instructional design standards are included in an organization’s larger standards, governance, and controls framework when such standards include a training component. Examples include standards related to safety, regulatory, legal, compliance and customer. In such cases, they are likely to be subject to third-party audits. In these more high-profile or high-stakes situations, failure to pass an audit may

298

Mastering the Instructional Design Process

EXHIBIT 17.1. GUIDING PRINCIPLES FOR INSTRUCTIONAL DESIGN GOVERNANCE Guiding Principles to Create Standards for ID Will this intervention help to achieve the organization’s goals? Explain:

Is it aligned with our organization’s values? Explain:

Will it create greater efficiencies or reduce redundancies? Explain:

Will it reduce costs? Explain:

Will it solve the intended problem? Explain:

Will it help to meet business needs? Explain:

Will the benefit to this intervention outweigh the costs? Explain:

Is there organizational support for it? Explain:

Is this intervention achievable? Explain:

What conclusions do you draw from the questions above?

Yes

No

Applying Business Skills to Managing the Instructional Design Function

299

lead to follow-up actions to remediate the problem or may cause financial or other penalties that can sometimes be severe or costly. These types of scenarios often lead to required standards as discussed previously. Sometimes, the standards transcend the organization itself and are based on industry-mandated practices. No, or little, discussion, debate, or consensus is necessary. What must be done, by whom, and how is specified in very black and white terms. The Instructional Designer Competencies: The Standards, Fourth Edition provides a comprehensive framework that can be a starting point for establishing standards of excellence. Given the wide-ranging standards outlined in this document, there will likely be a need to prioritize among them. Using various criteria can reduce the list and ultimately help choose which standards to adopt. Besides turning to The Standards as a potential source that could be considered, there are other ways to establish standards of excellence related to instructional design. A more organic approach is one alternative. Instructional designers themselves or managers could ask themselves “what could have a positive outcome on the organization if we were to establish a standard around it?” A list of potential areas could be generated and then narrowed using criteria such as the ones listed previously.

Implementing Standards A business case is an articulation of the financial and nonfinancial rationale for undertaking a new initiative, investment, or change effort intended to add value. A business case is often created to request support for resources, funding, or approval. Decision makers can review the business case and determine whether to support or approve it relative to other priorities or investments articulated in competing business cases. A business case may be highly formalized or it may be informal. Likewise, it may be very extensive or brief in length, breadth, and depth. “What’s the business case?” is an understandable question that any stakeholder or decision maker may ask when approached for funding or resources. The savvy instructional designer will be prepared if asked such a question and even head it off by bringing forth a business case before being asked. A business case may be required or desired for an individual instructional or noninstructional project or it may be necessary for establishing or changing an instructional design function. While significant time and energy may be needed to develop a compelling business case, doing so can

300

Mastering the Instructional Design Process

be very beneficial to the designer because it encourages diligent thinking, careful planning, and analysis, which can bring clarity and rigor prior to work commencing. A business case typically includes several key elements, but depending on the purpose and objectives, organizational expectations, and potential constraints, this may vary. The structure outlined below depicts the elements that may be contained in a comprehensive business case. • • • • • • • • • • • • • • • • • • • • • • • • •

Introduction Table of contents Executive summary Summary of business need Recommendation Next steps or action Overview Business need, problem, or opportunity Key business metrics Qualitative data Analysis Quantitative analysis of metrics Financial analysis Costs Quantitative and financial benefits Qualitative or intangible benefits Risk assessment and contingency plans Potential interventions or solutions Recommendation Decision Intervention or project purpose, scope, and timeline Resources required Key considerations Next steps Appendix

The level of rigor, depth, and time spent creating a business case should be matched to the situation at hand. The higher the financial and people resources being requested, the more time and rigor should be invested. Other factors such as the level of scrutiny expected, familiarity with the project or work proposed, severity of the problem or upside of the

Applying Business Skills to Managing the Instructional Design Function

301

opportunity, relative difficulty or ease to create the business case, political dynamics and skills, and ability in business case creation are a few of the many aspects that should be considered when thinking about developing of a business case. The goal of a business case is to present a picture of a potential project or initiative in an objective and compelling manner so that a sound business decision can be made regarding whether to support, fund, or resource the effort. The ability to create and present a strong business case is an important business skill that new, as well as, experienced instructional designers may be called upon to develop.

Leveraging Internal and External Resources to Accomplish Project Goals and Objectives An instructional designer rarely works alone and most often relies on a whole cast of characters to accomplish the goals and objectives of any design project. For this section, the term resources refers to people resources or to the general and specialized skills, capabilities, and capacities important to project success. Nonpeople resources are important as well and include financial, technological, physical, and capital resources, among others. Resources may be internal or external to the organization. Internal resources may come from the learning and development function or they may reside elsewhere in the organization. External resources are skills and capabilities that reside outside of the organization but that can be tapped to provide partial or more extensive support to the work. To leverage resources, they must first be identified and secured. Identifying resource needs involves examining the people who may be needed at every stage of the project. During the design phase an internal subject matter expert (SME) may be needed to help build content. Prior to deployment, a communications specialist may be needed to design a marketing and communication campaign to generate enthusiasm, interest, and participation. Facilitators may need to be secured from the business to deliver the program during implementation. During evaluation, a measurement expert with quantitative research skills may be needed to analyze data collected postintervention to determine impact on business results. These are just a few resources that may be needed during an instructional or noninstructional intervention. Exhibit 17.2 provides a list of potential resources needed during the instructional design process. This exhibit was presented in Chapter Sixteen (Table 16.1). We are including it again to show the various roles and resources.

302

Mastering the Instructional Design Process

EXHIBIT 17.2. PEOPLE-RESOURCES NEEDED DURING THE INSTRUCTIONAL DESIGN PROCESS Before the intervention

During the intervention

After the intervention

Subject Matter Experts

Provide information regarding content

Help to course-correct if needed during intervention

Communication Specialists

Create communication plan to identified stakeholders

Project Managers

Create timeline and rollout plan for intervention Collaborate with various stakeholders to bring resources together Complete needs assessment Create intervention objectives Design and develop curriculum Create evaluation plan Pilot the intervention Communicate with various stakeholders Support the instructional designer (design of exercises, predicts audience reactions, design room layout, technology needs, etc.) Provide graphic design support during the design and development of the intervention

Support evaluation and needed edits to curriculum Support learner application and follow up Support communication components of the evaluation plan Provide project management support for the evaluation plan

Aid in any course correction throughout the intervention

Provide participant support as they transition back to the work place Conduct post evaluation Report evaluation results

Facilitates intervention Provides one-on-one coaching

Support the instructional designer with the evaluation plan

Instructional Designers

Facilitators

Graphic Designers

Applying Business Skills to Managing the Instructional Design Function

303

EXHIBIT 17.2. PEOPLE-RESOURCES NEEDED DURING THE INSTRUCTIONAL DESIGN PROCESS, cont’d. Before the intervention

During the intervention

Technology Experts

Provides technology design and set up

Provides technology support

Business Leaders

Provides business perspective during intervention design Provides funding and resources Sponsors or markets the intervention Removes barriers to success

After the intervention

Removes any technology used during intervention Provides support (guest Reviews evaluation speaking, results coaching/mentoring Decide next steps participants) Intervenes with unexpected issues

Managing Instructional Design Talent Like any other critical group in an organization, instructional design talent must be managed effectively. Talent management is a systematic and systemic process to ensure the right skills and capabilities are available at the right time to contribute to helping the organization meet its important goals and enable current and future success (Rothwell, Jones, Kirby, and Loomis 2012). Talent management encompasses several dimensions from recruiting, performance management, development, succession, compensation, and others. Several of the more essential or foundational elements will be addressed. Most modern talent management frameworks are competency-based. Competencies are the important knowledge, skills, and abilities that contribute to successful performance in a job or role. For applying talent management to the field of instructional design and instructional designers, the competencies and standards in The IBSTPI Standards provides a model upon which talent management efforts can be centered, or at least provide a starting point. The final instructional design competency model may be tailored based on the specific needs, issues, and imperatives of the organization. Once finalized, competencies are embedded or integrated into the talent management elements being utilized in the organization. In this way,

304

Mastering the Instructional Design Process

they become the focal point by defining the end state. Talent acquisition, commonly known as recruiting, is a foundational strategic talent management process that involves sourcing, selecting, and hiring someone to fill a vacancy or open position in the organization. When competencies have been integrated into the talent acquisition process, a large part of the selection process will be geared toward ferreting out whether potential candidates possess, competencies deemed important. Selection activities will be geared toward finding and securing the candidate most proficient in the most important competencies so the likelihood of successful performance in the job, and culture fit in the organization is achieved. Recruiting Instructional Design Talent Recruiting is perhaps one of the most strategic parts of talent management and human resources functions in organizations. The reason is that identifying, selecting, and hiring the best talent at the right time for the right role helps to propel an organization forward in pursuit of its purpose and strategic objectives. Poor hiring decisions can lead to headaches, heartaches, underperformance, and sometimes the decision to separate with the person and start from scratch. Hiring the best talent possible, within budget, is typically much easier and effective than trying to motivate, engage, or develop an employee who should not have been hired. There’s an old adage that says “it’s easier to hire a squirrel than it is to train a turkey to climb a tree” which underscores the importance of hiring the right talent. Recruiting instructional design talent begins when a vacancy is identified either due to departure of an incumbent or a new position being created. A compensation range is typically determined using external benchmarking and internal salary analysis to ensure that strong candidates can be attracted and secured through a competitive pay package. Sourcing of candidates is done internally or through directing search efforts in the external marketplace to secure a candidate from outside the organization. Position descriptions are developed to describe the job requirements and typically include things like the number of years of experience, educational background, technical skills and knowledge, and the key interpersonal, communication, or leadership skills desired. Exhibit 17.3 is an example of a job position description for an instructional designer. Once potential candidates are identified and narrowed, often through resume review or phone screening done by search consultants or recruiters, interviews with the hiring manager and others are conducted. Others involved in the process may include recruiters, peers on the

Applying Business Skills to Managing the Instructional Design Function

305

EXHIBIT 17.3. JOB DESCRIPTION FOR AN INSTRUCTIONAL DESIGN CONSULTANT

Learning and Performance Solution Organizational Design Consultant Seattle, WA XYZ International partners with commercial clients to deliver professional services and technology-based solutions in the technology and financial services industries. The firm utilizes industry expertise and innovative analytics to produce high-quality products for our customers. Since 1983, XYZ has been serving clients at all levels. More than 5,000 employees serve these clients from more than 50 offices worldwide. Our website is at XYZ123.com. Our Learning and Performance Solutions group focuses on providing practical solutions to challenges in management, leadership, and organizational change for financial services clients. Join a growing team of 30 professionals who provide training, organizational development, change management, diversity and inclusion, and strategic performance improvement services to companies in the banking, analytics, and financial services industries. Job Description We have an immediate opening for a Senior Instructional Designer. The ideal candidate will have at least five years of experience in consulting and the full life cycle of the instructional systems design (ISD) process to facilitate knowledge transfer from classroom to the job. This person will manage at least two junior designers and graphics design associates. Key Responsibilities • Work with clients and subject matter experts to perform needs assessments to identify critical skills. • Recommend appropriate learning strategies and solutions to achieve desired outcomes. • Independently develop mid level and advanced level courses for both internal and external audiences. • Utilize multimedia technology and authoring tools to design and develop instructional materials including e-learning, classroom instruction, job aids, evaluations, and other performance support tools. • Acts as a liaison between various stakeholders including business leaders to determine needs of the course and appropriate method of delivery. • Manage junior staff including instructional designers and graphics professionals to review their products for quality, accuracy, and instructional soundness. • Develop and maintain style preferences and other instructional design guidance documents. • Manage instructional design projects to meet milestones and timelines to including pilot studies. • Conduct evaluations of participants, learning, and program success.

306

Mastering the Instructional Design Process

EXHIBIT 17.3. JOB DESCRIPTION FOR AN INSTRUCTIONAL DESIGN CONSULTANT, cont’d. Basic Qualifications • MS/MA in Instructional Design, Education, or related field • 5+ years of experience developing and mentoring junior staff in the development of instructor-led training, online training, job aids, and performance support tools • 8 years of experience working with subject matter experts and clients • 8 years of experience conducting detailed training and/or job/task analyses Preferred Skills/Experience • Experience in developing training in banking and/or financial services • Experience in one-on-one coaching leaders and managers • Experience using at least one rapid development tool (e.g., Presenter, Captivate, or Articulate) • Experience with project management software • Experience in consulting with clients Professional Skills • Strong oral and written communication skills • Strong analytical, problem-solving, and decision-making capabilities • Strong attention to detail • Ability to multitask in a fast-paced environment • Ability to work well both individually and in a team environment • Ability to prioritize multiple tasks while demonstrating the initiative to complete projects with minimum supervision • Ability to work with all levels of internal staff, as well as outside clients and vendors • Ability to be flexible to handle multiple priorities and to work limited overtime as necessary • Ability to travel up to 25 percent of time • Proficiency in MS Office (Word, PowerPoint, Outlook, Excel) • Sound business ethics, including the protection of proprietary and confidential information

learning team, customers, and others who will interact frequently with the designer and would have a stake in the selection process. Sometimes several rounds of interviews are conducted with candidates and may be one-on-one, done in pairs, or panel style with multiple people interviewing the candidate. Interviews may vary in duration with the standard length being 45–60 minutes.

Applying Business Skills to Managing the Instructional Design Function

307

When a candidate progresses to the interview stage (by phone, video conference, or in person), it would be expected that some of the interview questions would be based on competencies. Many organizations use a common approach known as behavioral interviewing. If behavioral interviews are used, those conducting the interview pose questions to the candidates based on selected competencies and the questions would be aimed at actual past experiences. The philosophy of behavioral interviewing is that the best predictor of future performance is past performance. Candidates are asked to describe a situation, what they did, and the outcome or result. If an interviewer was attempting to determine an instructional design candidate’s proficiency in dealing with conflict, the following could be asked “Tell me about a time when you led an instructional design project team that experienced some sort of conflict.” The candidate’s response would be framed as “Describe the situation when … ,” then “What you did in this situation,” and finally “The result or outcome (or what happened).” A strong response will describe the situation and the actions they took and what happened. This level of detail gives important clues about how the candidate has handled critical situations in the past, giving the interviewer a window into how he or she might perform in their organization. Or course, behavioral interviewing is no guarantee of a successful hire, but it does help to increase the chance of success. Similarly, having multiple people interview candidates can provide additional perspectives and increase the likelihood of uncovering a candidate’s strengths and weaknesses. Selection assessments, especially those that are research based and objective, can provide additional important insights and information about a potential candidate’s style, personality, and other attributes important to know before making a hiring decision. Onboarding Instructional Design Talent Once a selection decision is made and the candidate accepts the offer the onboarding process begins. Onboarding is an important talent management process that helps to ensure a new employee, or a current employee entering a new role, makes this transition smoothly and effectively. A poor onboarding experience can cause an otherwise highly enthusiastic employee’s motivation and engagement to plummet. When onboarding is done effectively, it helps the new person understand important aspects of the organization, culture, and role so they can become both productive and engaged rapidly. Many organizations have onboarding programs that help to achieve these objectives at both an enterprise level and at the

308

Mastering the Instructional Design Process

departmental level. Often, more generalized onboarding begins even before the candidate’s first day when they are physically on-the-job or even considered a full-time employee. Once the new joiner begins employment they may attend a new employee orientation program to provide general information about the organization, culture, benefits and other important information. As the new instructional designer progresses toward the department they will be part of, onboarding becomes much more specific and can include introductions to key people, an overview of department objectives, and access to key technology platforms used by the department. Onboarding culminates with a focus on the role the person will assume. Often conducted by the manager, the focus is on reviewing individual goals and objectives, setting expectations, discussing working styles and preferences, and assigning new or existing projects, tasks or work to be performed. Engaging and Retaining Instructional Design Talent The term engagement is defined as having the motivation to actively allocate personal resources toward tasks associated with a work role. (Kanfer 1990; Rich, LePine, and Crawford 2010). Engagement has been linked to performance and positive organizational and talent outcomes including retention, engagement, morale, job satisfaction, productivity, and performance. While not a talent management lever, engagement and retention is an outcome of effective talent management strategies and in some models is considered a key component. Determining the level of engagement, developing strategies to enhance it for instructional designers, and achieving these positive outcomes, is not dissimilar to how one would approach it for any other employee. High engagement is important to strive for with all employees, but given the direct correlation with retention and performance, it is especially critical to achieve with critical talent such as high performers, high potentials, and employees with highly valuable expertise or capabilities that the organization needs to achieve its objectives. Assessing and enhancing employee engagement is a leadership activity. Some organizations invest large amounts of financial and people resources to participate in surveys to measure employee engagement. The Corporate Executive Board can help organizations to gain insight into their employees’ perceptions of past events, current experiences, and future expectations through using their HRLC’s Employee Engagement Survey. Gallup provides another common assessment. They have identified 12 questions

Applying Business Skills to Managing the Instructional Design Function

309

that can be tracked over time to provide organizations insight into their employees’ level of engagement, as compared to other organizations. Such tools are used to determine overall engagement and also the key drivers of engagement. An engagement driver has been identified through statistical analysis to have a correlation to overall engagement. A common engagement driver across a wide variety of organizations and industries is career advancement. Other factors that can correlate, positively or negatively, with engagement include compensation, sense of teamwork and cooperation, and manager relationship. Besides measuring and attempting to improve engagement at a more macro or organizational level, managers are well positioned to ascertain and take actions to improve engagement within their teams. Even if armed with department-level engagement data such as the type described previously, it’s important to understand engagement at an individual level. This often happens on a more informal basis through one-on-one conversations with direct reports. Being highly attuned to the mindset, morale, and attitude of employees comes more naturally to some progressive managers. Others, who may be less dialed in to their people, may need to work more diligently. In either case, focusing on employee engagement is an intentional and purposeful pursuit (Rothwell, Alzhahmi, Baumgardner, Buchko, Kim, Myers, and Sherwani 2014).

Developing Instructional Design Talent To avoid the “cobbler’s children” syndrome, a robust development strategy for instructional designers is an important part of an effective talent management strategy. Many instructional designers have been formally educated in instructional design methodology and skills, through a certification program or a graduate degree. ATD (Association for Talent Development) has created a certificate program for instructional designers to refresh their instructional design skills with more updated technology based techniques. Several well-known colleges and universities have masters and even doctoral programs focused exclusively on educating instructional designers (Penn State University, University of Wisconsin, George Washington University, Syracuse, Boise State, Florida State, San Diego State are a few). Obtaining a degree or certification is an excellent way for a current practitioner to enhance their instructional design skills. It’s also a means by which a person can enter into the field as a new practitioner.

310

Mastering the Instructional Design Process

Individual Development Planning Most talent management frameworks include some sort of development planning component in which an individual creates an action plan in partnership with his or her direct manager. A development plan is typically based on formal and informal input from various sources, such as performance management feedback, manager observations and input, peer or customer feedback, or formal assessments that the person may have participated in. These insights help the designer surface and articulate his or her top strengths and weaknesses, which are often given more palatable terms like opportunities or areas of improvement. Through this heightened sense of self-awareness of strengths and opportunity areas combined with clear understanding of performance objectives, development goals are crafted. These goal statements articulate the outcomes or focus of the development efforts. Strong development plans go beyond stating the goal and often include other important elements such as milestones toward the goal, metrics or measures of success and support, or resources needed. While the development plan focuses on the individual, ideally others are involved in supporting the person. First and foremost is the manager of the individual (Rothwell and Chee 2015). The direct manager is well positioned to provide feedback that can help in the up-front efforts to raise self-awareness and create the plan. Importantly, the manager can provide ongoing observations and feedback and support that will help ensure success. Support may come in encouragement and more tangible support such as providing time, resources, and opportunities essential to achievement of development goals. Professional Skills Development Beyond developing the more “technical” skills, knowledge, and tasks essential for successful performance as an instructional designer, developing nontechnical skills is equally important. These so-called “soft” skills can include a wide variety of areas such as communication, leadership, project management, negotiation, and interpersonal effectiveness skills. Many organizations offer internally conducted courses, workshops, and e-learning offerings that cover topics such as these. Most involve an in-person or classroom-based component so participants can not only learn about the skills, but also practice them and receive feedback as they work to build their capabilities.

Applying Business Skills to Managing the Instructional Design Function

311

Besides internal offerings, many external providers exist for instructional designers and other professionals to attend these types of skill building opportunities. They may vary in length from one day to five or more depending on the depth of coverage. Some are held in an “open enrollment” format where participants from different organizations attend. Private or closed sessions are attended by participants from the same organization, often with the external vendor conducting the session onsite. Participants attending external seminars pay a set amount to attend and the prices can range from one hundred dollars or less to thousands depending on the program, the provider, and the length. Many colleges, including community colleges, and universities offer a variety of higher-end executive education programs as well as shorter workforce development programs. Seminar and workshop providers such as The American Management Association (www.amanet.org/), Management Concepts (www.managementconcepts.com/), the Institute for Management Studies (www.ims-online.com/), online provider cornell (www.ecornell.com/), and many others offer hundreds of in-person and e-learning programs to help designers build their skills and capabilities and contribute to effective performance. Besides leadership and soft skills professional development, many designers engage in ongoing development to keep their instructional design skills sharp or to build new skills and knowledge to improve their job performance. Newer instructional designers may benefit from short courses or certifications to build their instructional design skills. Experienced designers may benefit from refresher courses and from courses that go deeper and specialize in particular topics or skill areas such as advanced instructional methodologies, e-learning design, multimedia strategies, and other technical skills. Some of the more common sources for instructional design-related training are the Association of Talent Development (www.td.org), Capella University (www.capella.org), and Kaplan University (kaplanuniversity.org). Conferences are another means by which amateur and experienced designers can build their instructional design skills. Many organizations offer conferences where instructional design practitioners at all levels congregate to learn the latest in instructional design techniques. Some of these include the International Society for Performance Improvement (www.ispi .org), the Association for Talent Development (https://www.td.org), the Association for Educational Communications and Technology (www.aect .org), and the Masie Center (www.masie.com).

312

Mastering the Instructional Design Process

Career Development and Progression Another key area of talent management relates to career development and progression. Career development in organizations helps employees identify their goals and aspirations, identify strengths and opportunity areas, set career goals, and develop plans to achieve their goals. There are many parallels between career planning and development planning mentioned previously. One distinction is that career development is longer term whereas development planning is typically shorter term in pursuit of nearer term performance goals. Career development and progression for instructional design may be less open when compared to general career development for other professions. One way to think about career development in a somewhat defined field such as instructional design is to look at the opportunities to progress both vertically or upward in the organization and horizontally or laterally. Vertical career progression involves advancing from a lower-level to a higher-level role, typically with increased scope, responsibility, expectations, and compensation. There may be several tiers of instructional designers in an organization. For simplicity’s sake, let’s assume that an organization has three tiers of instructional designers. An Instructional Designer Level 1 may support projects being led by a more senior designer. An Instructional Designer Level 2 may lead smaller-scale projects and participate in more complex large-scale projects under the guidance of a more senior designer. An Instructional Designer Level 3 may lead large-scale projects and provide guidance to less experienced designers. It’s easy to imagine a vertical progression through these three levels. Lateral career progression involves moving into a new or different area at the same level with comparable scope, role expectations, and salary. When individuals engage in lateral career moves, it may not appear to be progression, but it can help them to build broader skillsets that ultimately help them prepare to assume higher-level roles. Lateral career development can also be helpful in entering into new areas without having all of the immediate skills needed. It can also help to build a broader perspective, which is important as one ascends to higher levels within an organization. The likelihood of short- and long-term success for lateral moves is enhanced if the individual involved is interested and motivated for both the immediate lateral move and the future vertical opportunity. If someone loves their current role or does not aspire to advance, then forcing it to occur may backfire and lead to decreased morale and attrition.

Applying Business Skills to Managing the Instructional Design Function

313

Marketing and Communicating about Instructional Design As a cost center, or service provider, within an organization (rather than a profit center), an important role of an instructional design function or team is to market and communicate the products it creates and the services it provides. There are several purposes to these efforts. One is to make the work of the instructional design team more known within the broader organization so awareness is heightened. As awareness is increased, customers and potential customers will have a better understanding of what is available and how to access it. Another purpose is to demonstrate the value and contributions the instructional design team has on the organization. Various marketing and communication channels are available. Also, many media may be used. Often, the services and products of the instructional design team are marketed and communicated as part of a broader plan. This is the case because in most organizations, instructional design is a subset of a larger team, such as enterprise learning or talent management, and not large enough to warrant stand-alone marketing and communication efforts. As an example, an organization may offer a catalog of learning opportunities such as classroom courses and online learning that are available. Such catalogs may be print-based or electronically delivered or posted for on-demand viewing and include the offerings, a brief description of the target audience, content and learning objectives, and dates and registration information. A resource such as this, produced by the broader learning team, can easily be leveraged and include information about instructional design services. Besides embedding information about instructional design products and services in other marketing and communication vehicles, stand-alone resources can be created and distributed. The purpose of such communications is similar to the intent of embedded tools like the one described above. Before the actual tangible marketing or communication tool is built, a plan should be created. A marketing and communication plan should include an objective for the plan, communication points, the audience, the forum in which the communication points will be presented, and a timeframe. Once the plan is created, and approved, the plan is put in motion. Executing the plan involves developing the material, often called collateral, and distributing it using various media and channels. Sometimes marketing pieces are simple and may be created using various software packages like PowerPoint or Word. These may contain descriptions of

314

Mastering the Instructional Design Process

instructional design products and services and can be sent electronically to clients or potential clients, or they can be printed and sent or reviewed with stakeholders during in-person or virtual meetings. Other marketing collateral may be more sophisticated and professionally designed and developed using custom-designed graphics, images, and animation. Sometimes learning or instructional design teams develop a comprehensive set of marketing and communication material that is modularized. Various modules or components can be selected and used based on the client and the objective or the entire package can be used. Another practice to increase flexibility is to create collateral using multiple media formats. An overview of the instructional design team may be created using a presentation software package. This could be printed and distributed in hard copy or it could be attached to an e-mail and sent to customers electronically. It could also be converted into an animated self-running presentation with voiceover that can be sent to customers or posted on a team website, where potential customers will visit or where they can be directed to for more information.

Navigating Organizational Culture, Structure, Politics, and Processes Seeking and securing of management and stakeholder support occurs in the culture, structure, politics, and processes of the organization. These elements form the context within which the designer must operate to not only survive but to flourish. Navigating culture and politics is not exclusive to instructional designers working in large, complex, bureaucratic organizations. Likewise, it’s not exclusive to those working in certain types or organizations or industries. Smaller organizations have their unique set of challenges, like fierce competition for scarce resources. Family-owned or private organizations, organizations that may struggle, those enjoying dramatic growth, and those in highly regulated industries all face their own set of challenges that the instructional designer must understand and work effectively within. No one is immune from the context within which they operate, but it is possible to succeed through understanding, adapting, and knowing how to navigate and lead effectively. The culture of the organization cannot be ignored. Understanding the spoken and unspoken norms of behavior and how people operate can have an enormous impact on the success of the instructional design group. Sometimes aspects of culture are present across an entire organization.

Applying Business Skills to Managing the Instructional Design Function

315

Other times cultural characteristics may exist within a subset of an organization such as the environment in a particular department, team, or business area. Often when someone enters an organization, things are detected rather quickly about organizational norms because they are explained as expectations to the new person. Other aspects of culture may be more difficult to detect or fully grasp because they are subtle or may not exist widely across an organization. Asking ones manager, peers, mentors, or others is one way to get a handle on the culture. A new joiner to an organization, department, work group, or team could ask people questions like “How would you describe the culture here?,” “What should I know about how things work around here?,” “Do you have any advice for me as I start working in this organization or environment?,” “What does it take to successfully navigate our culture?,” “What gets people in ‘trouble’ here?,” and so forth. Another helpful technique for a design professional is to meet with a variety of people and ask “What do you like most about the culture here?” and “What do you like least?” Collecting the words or phrases people use in response to these two questions and surfacing common themes can give the designer a high-level sense of the culture. Once a grasp of the current culture is formed, an individual can modify and adapt behaviors to be effective. If an organization is highly relationship oriented, it may be important for someone who is introverted to contact people, get to know them professionally, and personally communicate regularly with them. For a design professional to confine him- or herself to a desk and avoid interactions would contravene this cultural norm and could eventually cause the person to be isolated from the mainstream and perhaps even ejected from the organization. Not all individuals should feel pressured to adapt to or assimilate into the prevailing culture. As more organizations recognize the importance and benefits of being more inclusive, people different from others, in traditional and nontraditional ways, should be welcomed because it adds to the richness and diversity of the organization. Still, certain norms of operating may be important to understand and essential to adapt to. Besides fostering an environment of inclusive diversity where differences are welcomed, some individuals are asked to join an organization to disrupt certain aspects of the culture. A highly creative individual may be hired to bring greater innovation and to bring a fresh perspective to a team that is more risk averse or status quo. Depending on change needed from the current state to succeed, the change agent must be skilled and balance understanding, adapting to and pushing organizational change.

316

Mastering the Instructional Design Process

Having support in these efforts is equally important so the person is not operating in a vacuum or rejected by those who may be uninterested or ill equipped to change. Similar to culture, organizational politics reflect the norms of behavior and often explains how work gets done and by whom. Related to career development, discussed earlier, sometimes politics can come in to play regarding who advances in the company and how. Politics is often viewed negatively by many in organizations and especially by those who have been “burned” or affected negatively and suspect that politics may have been at fault. No matter how much one would either like them to not exist or pretend they don’t, politics are as much of a fact of organizational life as culture. There are various degrees to which people enjoy or disdain politics and also degrees to which people get actively involved in them, by choice or not. Some attempt to distance themselves from politics as much as possible while others make a career out of playing politics. Similar to culture, mastering political dynamics in organizations begins with recognizing their existence and then working to understand how they operate and, finally, to the extent possible, attempting to work with, through, or around those dynamics to be effective. Calling out or confronting politics head on can be dangerous business due to the sensitivities and personal attachments associated with doing so. A more palatable take on organizational politics is to engage in a kind of organizational savvy. Instructional designers must be savvy in understanding the political dynamics that exist in an organization—especially any that have direct impact on the work of the project, team, or other key stakeholders.

CHAPTER EIGHTEEN

MANAGING PARTNERSHIPS AND COLLABORATIVE RELATIONSHIPS

T

he previous chapter introduced the first competency related to managing the instructional design function. It focused on applying business skills and was directed largely at the instructional designers or the immediate team. This chapter looks outward from the instructional design team and addresses important skills associated with managing partnerships and collaborative relationships with other stakeholders. Instructional design is not a solitary pursuit. Instead it requires relations hips with many people. These relationships may be internal to the organization or can extend to partnerships with external stakeholders. When designing interventions, interaction with others is not always easy because it may require people to think and act in new ways and sometimes function under time pressures and high stress. The Standards (Koszalka, Russ-Eft, and Reiser 2013, 65) states that, “professionals who are proficient in this competency recognize that instructional design work is accomplished with and through other people. These professionals consider all the individuals and groups interested and/or involved in the work.” This chapter is about the critically important soft skill of interacting effectively with others. Someone could be the most knowledgeable and technically proficient instructional designer in the world and fail miserably if he or she cannot interact, and build productive relationships, with others. The soft skills can 317

318

Mastering the Instructional Design Process

sometimes be the most difficult to master and can derail both person and project. According to The Standards, “this is a managerial competency and it includes four performance statements, one is advanced and three are managerial” (Koszalka, Russ-Eft, and Reiser 2013, 65). Instructional designers who demonstrate this competency can: “(a) identify stakeholders and the nature of their involvement; (b) build and promote effective relationships between the design team and stakeholders; (c) manage cross functional teams; and (d) conduct project reviews with design team members and stakeholders” (Koszalka, Russ-Eft, and Reiser 2013, 65). Interpersonal skills are crucial to success in the instructional design field. In this chapter, after defining and describing stakeholders, we will discuss how instructional designers establish rapport, ask questions, listen actively, deal with conflict keep people on track, and secure commitment. Effective interpersonal interaction leads to strong partnerships and enhanced collaboration toward shared goals and results.

Identifying and Engaging Stakeholders Instructional design work might involve many stakeholders. Stakeholders include clients, sponsors, champions, learners, managers, partners, and others having a vested interest in instructional outcomes. Different stakeholders have varying needs, expectations, and levels of involvement in the ultimate success, or failure, of the work. Regardless of the roles of individual personalities and their desires or expectations, instructional designers must establish trust and rapport, work effectively with, and ultimately do their work with, through, and for these stakeholders. Successful performance depends not only on the technical aspects of instructional design, but also the nontechnical interpersonal skills. It’s often the ability to work effectively with and through others using communication, influence, and leadership skills that leads to success. Inability to exhibit such soft skills can lead to project hiccups and underperformance in the best case and, at worst, the demise of the project and even career derailment. So, an important skill instructional designers must learn is identifying stakeholders, determining their level of support and involvement, and engaging them effectively. Begin the stakeholder analysis process by brainstorming a list of all the potential stakeholders who may be involved in a design project, their roles, and the nature of their involvement. To illustrate, Alexandra Deer is

Managing Partnerships and Collaborative Relationships

319

a senior executive and her role is multifaceted and includes funding the project, signing off on the final deliverable, and freeing up resources to work on the project. There are clear reasons to consider her as an important stakeholder. Once a complete list of stakeholders is generated, the analysis moves toward assessing each person against a variety of potential criteria. This helps to determine both the importance of the stakeholder and the energy that may need to be invested in engaging with that stakeholder. One example of such a criteria is the power or influence the person has on the project. One stakeholder may wield a great deal of power while another holds little or none. Often simple ratings like low, moderate, or high, or a color scheme like red, yellow, or green can be used. Another common criteria used in a stakeholder analysis is resistance, or conversely the level of support for someone may have to the project or change. A five-point scale for resistance might be highly resistant, resistant, neutral, supportive, and highly supportive. The next step is to determine the action that must be taken to engage each stakeholder and move them toward greater levels of engagement and support. This is sometimes referred to as stakeholder management, whereby purposeful and customized engagement strategies are defined and executed. If a key stakeholder such as an important SME is assessed as being “highly resistant” to the project because of a perceived loss of knowledge and control, a targeted strategy can be devised whereby this person is engaged to first understand the root causes of his or her concerns and then to allay them. Before the interaction, decide who will approach the stakeholder, the engagement tactics employed, key messages to be conveyed, and other means by which the person will be enrolled. In this manner, the effort is geared toward moving the person from being resistant to being neutral or supportive in his or her position. The Importance of Rapport Relationships are critical to success in instructional design, and effective relationships depend on rapport. Establishing rapport is essential for instructional designers. Rapport, in its simplest sense, can be understood to mean interpersonal trust. Hence, establishing rapport means creating a trusting relationship with another person or group of people. It is perhaps best understood as synonymous with effective interaction, meaning the interpersonal skills leading to mutual trust.

320

Mastering the Instructional Design Process

Perhaps the most important single element in working as an instructional designer is this ability to inspire (and maintain) trust. Trust has been the focus of much attention in recent years. For instance, McLagan (2003) found it to be key to a “change-capable” organization. Successful CEOs are those who can establish and keep employee trust (Finley 2002). Effective executive coaches inspire client trust (David 2002). Work on virtual teams requires trust (Guillot 2002), and trust is essential in groups demonstrating emotional intelligence (Druskat and Wolff 2001). But virtual working arrangements and volatile working conditions, prevalent in so many workplaces today, undermine trust because face-to-face relationships are essential to it (Prusak and Cohen 2001). Kirkpatrick (1978, 46), in his description, listed the key conditions of rapport: (1) There is mutual respect between sender and receiver; (2) Friendly relationships exist between sender and receiver; (3) The sender encourages questions and feedback from the receiver; (4) The receiver doesn’t hesitate to say, “I don’t understand”; (5) When [the] sender says “Do you have any questions?,” the receiver feels free to ask questions without fear of being embarrassed or ridiculed; (6) The sender is willing to accept responsibility for the receiver’s understanding or lack of understanding; and (7) The sender compliments [the] receiver for understanding and blames self if the receiver misunderstands.

To these conditions we can add several listed in Foshay, Silber, and Westgaard (1986, 92) that are still relevant today: “(1) The dialogue should continue as long as the instructional designer wants, (2) information is not withheld from or by the instructional designer, (3) false assumptions about the instructional designer are not made, and (4) information given to the instructional designer is not changed to meet the individual’s or group’s assumptions about what kind of person the instructional designer is.” How to Establish Rapport But how are these conditions established? The instructional designer’s affiliation is the first consideration in creating and maintaining rapport. Conditions for establishing rapport differ, depending on whether you are an instructional designer working as an internal or external consultant. Insiders, working as internal consultants, are employed by the same

Managing Partnerships and Collaborative Relationships

321

organization as their potential clients. They have usually established a reputation by which group members may predict their behavior. However, outsiders working as external consultants are not employed by the same organization. Instructional designers who work within organizations sometimes enjoy an advantage over outsiders in that they may have more understanding of the organization’s culture and the key beliefs and values of its leaders. They can have unique insight into the causes and ramifications of a problem. Yet they may also experience difficulties in initiating contact with others, since insiders are so familiar that their expertise may not be fully appreciated, they may lack authority or access within the organization’s chain of command, or they may have experienced past problems with others so their ability to help is compromised. Instructional designers working as external consultants, however, may not have interacted with members of the client organization’s management before and may lack social ties. And members of the organization have had no experience—as they may have had with insiders—by which to gauge the instructional designer’s behavior or assess how the designer may interact with others. Yet instructional designers who work as external consultants enjoy certain advantages: they are often accorded expert status; they might enjoy a special reporting relationship with or access to top managers; and they might look at a problem with a fresh perspective, one not colored by in-house politics or organizational traditions and culture. The outsider sometimes experiences difficulties in establishing credibility. Outsiders must therefore work to establish, to the satisfaction of their clients and other important stakeholders, that they are trustworthy, knowledgeable, and diligent. By trustworthy, we mean that instructional designers must demonstrate that they are not mere pawns of top managers. By knowledgeable, we mean they must demonstrate enough knowledge about the organization, industry, and problem that they are viewed as credible. By diligent, we mean that instructional designers must be perceived by insiders as capable of researching problems in all their complexity and as capable of following up or implementing the solutions they propose even if there is resistance along the way. To establish rapport when working as an outsider, instructional designers must demonstrate thoughtfulness, a willingness to listen to what others have to say, and the ability to function within the norms of the organization’s culture. The situation is the second issue affecting interaction. Instructional designers typically initiate relationships with potential clients under two

322

Mastering the Instructional Design Process

possible sets of circumstances. In the first situation, help is requested by a prospective client, and instructional designers are asked to research a problem, assess needs, or otherwise investigate and take action. They do not initiate the relationship. They are asked for help, and they meet with a representative or group of representatives from the organization requesting that help. In the second situation, instructional designers request help from others. They then initiate the interaction and the relationship. Others are asked for information or for permission to obtain information. Both situations may be encountered in the same instructional design project. For instance, top managers or key middle managers may request assistance to analyze a human performance problem or design instruction. In conducting performance analysis, the instructional designer may also need to initiate contact with others in the organization to obtain information. The basic steps in establishing and maintaining rapport are outlined in Table 18.1. Study that table carefully. As you do so, you may wish to consider that establishing good rapport has important cross-cultural implications. As Odenwald (1993, 44) notes, “The strong task orientation of Western culture does not play well in many Asian cultures that regard establishing good rapport as the first order of business.” Instructional designers should also remain vigilant to the importance of rapport in some cultures—and adapt accordingly as project requirements necessitate.

Building Relations between Instructional Designers and Individuals or Group Members What is the nature of the interaction? How easily can instructional designers explain to others the tasks to be performed during their projects? How clear-cut is the range of possible strategies that can identify or address human performance problems? How easily can instructional design decisions be justified? When the nature of the activities to be performed in the interaction is easily understood, the range of possible strategies is limited, and instructional designers can justify their actions and decisions without too much difficulty, the situation is clear and structured. When activities are difficult to explain, the range of solutions is not limited, and instructional designers find it difficult to justify their actions and decisions, the situation is vague and unstructured. When the situation is structured, instructional designers should find interactions easy enough. They only need to explain their purpose and

Managing Partnerships and Collaborative Relationships

323

TABLE 18.1. A MODEL FOR SELECTING TECHNIQUES TO ESTABLISH AND MAINTAIN RAPPORT IN INSTRUCTIONAL DESIGN PROJECTS Affiliation Situation

Insider

Outsider

Request Initiated by Others

Before initial meeting Ask for advance information about the problem. Research the in-house politics of the problem, if possible. Research the person who requested assistance, if possible. Dress to make a good first impression. Prepare some questions in advance.

Before initial meeting Ask for advance information about the organization and problem. Find out what you can. Dress to make a good impression. Prepare some questions in advance. Prepare biosketch and list of references who can provide information about your skills/abilities.

During initial meeting Allow the initiator of the meeting to set agenda. Take notes. Demonstrate attending skills. Ask “how can I help?” Determine purpose of meeting, nature of help required. Clarify your reporting relationship. Ask key questions. Clarify next steps.

During initial meeting Allow the initiator of the meeting to set agenda. Take notes. Ask “how can I help?” Determine purpose of meeting and nature of help required. Clarify your reporting relationship. Ask key questions. Clarify next steps.

After initial meeting Follow up, summarizing help requested and next steps.

After initial meeting Follow up, summarizing help requested and next steps.

In subsequent interaction In subsequent interaction Demonstrate thoughtfulness, keeping Demonstrate thoughtfulness, keeping others informed. others informed. Request Initiated by Instructional Designer

Before initial meeting Clarify protocol for contacting people and follow it. Make advance contact, clarifying nature of request. Try to arrange for another person in the organization to contact the individual, requesting his or her cooperation (when appropriate). Dress to make a good first impression. Prepare questions in advance. Prepare an agenda for the meeting in advance.

Before initial meeting Clarify protocol for contacting people and follow it. Make advance contact, clarifying nature of request and who has approved the meeting. Ask your “contact” in the organization to help arrange the meeting, when appropriate, to help lay the groundwork for cooperation and show evidence that the request for information has been approved by key managers in the organization.

324

Mastering the Instructional Design Process

TABLE 18.1. A MODEL FOR SELECTING TECHNIQUES TO ESTABLISH AND MAINTAIN RAPPORT IN INSTRUCTIONAL DESIGN PROJECTS, cont’d. Affiliation Situation Request Initiated by Instructional Designer (cont’d.)

Insider

Outsider Dress to make a good first impression. Prepare questions in advance. Prepare an agenda for the meeting in advance.

During initial meeting Run the meeting. Begin the meeting with small talk to set the individual at ease. Clarify who you are, where you come from, what you want, why you want it, how the information you request will be used, and who will see the results of any investigation you conduct. Establish your own credibility. Listen actively. Make your request specific and (if possible) show how providing information could benefit the individual to whom the request is being made. Encourage participation by the individual who will provide the information. Clarify next steps.

During initial meeting Run the meeting. Begin the meeting with small talk to set the individual at ease. Clarify who you are, where you come from, what you want, why you want it, how the information you request will be used, and who will see the results of any investigation you conduct. Establish your own credibility. Listen actively. Make your request specific and (if possible) show how providing information could benefit the individual to whom the request is being made. Encourage participation by the individual who will provide the information. Clarify next steps.

After initial meeting After initial meeting Thank the individual for his or her time Thank the individual for his or her time and effort. and effort. Summarize the meeting and next Summarize the meeting and next steps. steps. In subsequent interaction Remain thoughtful and considerate of others’ viewpoints and feelings.

In subsequent interaction Remain thoughtful and considerate of others’ viewpoints and feelings.

describe what steps they must take to collect information or find a solution. Unstructured or ambiguous situations are more difficult. In those cases, they should try to find an “idea champion” from the organization to help reassure those who may be concerned about the project and to function

Managing Partnerships and Collaborative Relationships

325

as a liaison with others. To reduce misunderstandings that could destroy rapport, instructional designers should take pains to brief managers at the outset of the project about their need for information, the steps in the project, and key assumptions underlying those steps. They can facilitate decisions and interaction. Much has been written about the importance of facilitation (Bentley 1994; Driskell, Olmstead, and Salas 1993; Kinlaw 1993; Schwarz 1994). Instructional Designers’ Position Power Position power refers to the ability, perceived or real, to exact obedience or compliance from others. When instructional designers begin an assignment with full support from top managers and easy access to them, their position power is said to be high; when they begin an assignment without full support from top managers or without access to them, their position power is said to be low. When position power is high, ask that a top manager or sponsor send out an e-mail or other communication to solicit support and cooperation during data collection. (Instructional designers may have to draft the message themselves for the top manager to send.) That should provide an adequate introduction to those who must be contacted within the organization. When position power is low, base information requests on the problem itself. Explain why the information is needed and how it can help solve the problem or prepare instruction. Point out the problem, state what information needs have to be met to solve it, and ask others for their cooperation. Use influence effectively to intervene in the inner workings of the organization (Tosti and Jackson 1992). The Power of Asking Questions Instructional designers should be able to “ask questions of individuals or groups so that they (1) gather all the information that is required for their purpose; (2) gather the information accurately; [and] (3) phrase and sequence questions so the individual or groups provide the information they have” (Foshay, Silber, and Westgaard 1986, 94). Questions are powerful tools for data gathering. They can shed new light on perceptions of problems, people, or events. Instructional designers may use questions to collect information about existing human performance problems, identify performance criteria or managerial expectations, pinpoint causes of performance problems, and determine the significance of those

326

Mastering the Instructional Design Process

problems. They can also use questions to consider possible solutions to address those problems, select one or more solutions, anticipate negative side effects that may stem from implementing solutions, establish goals for learners, test learners’ knowledge, manage classroom instruction, and for other purposes. Questions may be categorized in two ways. First, they may be open or closed. Second, they can be externally or internally focused. Open questions invite people to talk; closed questions shut off or redirect responses by yielding “yes” or “no” types of responses. Open questions begin with words like who, what, when, where, why, and how. Instructional designers may ask open questions such as “When did you first notice this performance problem?” “Who is affected by the problem?” “Where is it most and least evident?” Open questions can also begin with such words as could or would, as in the question, “Could you tell me a little more about … ?” Questions beginning with “could” prompt the most talking and offer the fewest number of clues about what the interviewer is looking for in an answer. Use open questions to explore and investigate problems, probe what others have said, and prompt creative thinking by learners. Closed questions begin with such words as is, are, was, were, do, did, have, and has. Instructional designers can ask closed questions such as these: “Is that an accurate description?” “Was that always the work standard?” “Do you have ideas about the cause of the problem?” “Have you tried to investigate this problem in greater depth?” Use closed questions to guide a conversation, by verifying information or tactfully shutting off further commentary. Externally focused questions are directed to conditions external to the person; internally focused questions are directed to conditions internal to the person. Externally focused questions are appropriate for collecting objective or factual information. Examples include any of the following questions: How would you describe the human performance problem your organization is experiencing? When was it first noticed? In what locations is it most apparent? Use externally focused questions to collect descriptive information. However, internally focused questions are appropriate for assessing attitudes, opinions, beliefs, and perceptions. Examples include any of the following: How do you feel about this problem? What do you think others feel about this problem? What is your perception of this problem’s cause or causes? Use internally focused questions to collect interpretive information. Another aspect of questioning centers on the tone of questions. With the emergence of appreciative inquiry as a new trend in organization development, many practitioners are rethinking their approaches. Instead of focusing around “solving problems” and “eliminating gaps,” appreciative inquiry centers on discovering what is working well and

Managing Partnerships and Collaborative Relationships

327

strengths to build on them. Questions based around this approach focus on the positive—what works, what makes people feel good, what strengths an organization can leverage, and so forth. Whitney, Cooperrider, Trosten-Bloom, and Kaplin (2001) have assembled a book full of questions that can be used along these lines. Listening Actively Instructional designers should be able to “listen to individuals or groups so that they gather sufficient information for the purpose, the individuals or groups feel listened to, and the individuals or groups continue to provide information for as long as you want them to” (Foshay, Silber, and Westgaard 1986, 96). Instructional designers should distinguish projects appropriate for instructional design from those that are not, conduct needs assessment, and carry out many other activities. Listening may require at least 48 percent of project time. However, when people are not trained to listen, their listening efficiency can dip to as low as 25 percent of a complete message (Nichols 1957). Part of the problem is that most people speak at a rate of about 125 words per minute, yet an average listener can think at a rate of about 400 words per minute. Hence, significant time can be wasted as one person listens to another. It is easy to fall into the trap of listening passively or planning what to say next rather than hearing what others are saying. However, listening should be an active endeavor, one in which the listener devotes as much attention to the feeling (emotional) components as to the content (meaning) components of the message. There are several keys to active listening. First, instructional designers should focus on what is being said on more than one level. They should ask themselves not only, What does the speaker mean? but also, How does the speaker feel about the subject at hand? Body language, tone of voice, and any other clues to meaning and feeling should be noted. Second, instructional designers should work at listening. They should ask questions, show interest, and use body language that encourages rather than discourages speakers. The key is to remain self-conscious, aware of how a listener’s actions influence speakers, while simultaneously focusing attention on a speaker’s content and feelings.

Managing Cross-Functional Teams Instructional designers often are parts of cross-functional project teams and sometimes are the leaders of those teams. Real differences of opinion can arise about instructional design projects. Personality conflicts can arise

328

Mastering the Instructional Design Process

between members of an instructional design team and between instructional designers and operating managers. As organizations have become more global, virtual teaming has arisen as a common way of working. The potential for conflict can be heightened when geographic distance, cultural differences, and inability to read body language is introduced.

How Does Conflict Arise? Conflict arises whenever individuals or groups disagree about philosophy, values, goals, measurement methods, or results. It also stems from differences in personal styles, communication problems, competition, association, interdependence, expectations, and change. As Hensey (1983, 52) points out, “conflict is often a result of changes, actual or perceived, and conflict is a very legitimate way of managing change, though not the only way. Planning, collaboration, problem-solving, and coexistence are some other ways of dealing with change.” Whenever people work together, undergo change, or experience interdependence, the potential for friction exists, and instructional designers work with many kinds of people: colleagues, operating managers, media production people, learners, learners’ supervisors, and others. Naturally, change and learning are synonymous, and instructional design projects typically require interdependence among team members.

How Should Conflict Be Managed? Before attempting to manage friction, instructional designers should first clarify their assumptions about it. Second, they should try to determine its cause. They should then apply one of many available approaches to manage it. The starting point for managing conflict is to clarify assumptions about it. Many people view any disagreement as something to be avoided. “Conflict,” as Baker and Morgan (1989, 151) point out in a still-relevant passage, “is often viewed negatively, although it is neither good nor bad in itself. If properly handled, conflict can become a positive source of energy and creativity; if mishandled, it can become dysfunctional, draining energy and reducing both personal and organizational effectiveness.” Each instructional designer should begin by clarifying his or her own views about conflict. If that is difficult, then he or she should think back to the last time in which he or she observed, or was a party to, a disagreement.

Managing Partnerships and Collaborative Relationships

329

It could have been a disagreement with a supervisor, coworker, team member, or even a spouse or friend. What feelings did it evoke? How was the conflict handled? Were any of the following ineffective conflict resolution strategies used to cope with it? • Moralizing (My way is right!) • Submitting (I’ll give up just to keep the peace, even though I still think this is wrong.) • Denying (Maybe we don’t disagree after all.) • Coercing (You better do it my way or else.) • Bribing (If you do it my way this time, I’ll see to it that you get your way on something else.) Reflect on past actions in conflict situations at work or at some other location. Determine what assumptions were made about the conflict. Was it handled as though it were best avoided? If so, rethink how it was handled, realizing that conflict is a natural part of social life and can be a stimulus for new ideas and strengthened relationships. The second step in managing conflict is to identify its causes. Apply essentially the same techniques used in performance analysis. Never be misled by symptoms alone. Symptoms of conflict may include arguments between people, name calling, malicious gossiping, the formation of cliques among team members, absenteeism, turnover, or even sabotage or workplace violence. These problems result from more than simple personality conflicts and may reflect much deeper causes stemming from differing philosophies, values, goals, or work methods. Consider the following questions: When did the conflict first appear? What are its consequences? Who is involved? Is it a difference between individuals or groups? What do the conflicting parties believe the causes of the conflict to be? Probe for answers, just as you would do in performance analysis. The third step in managing conflict is to apply a conflict resolution approach. Use collaboration, which means working with others to find a mutually satisfactory, if not mutually beneficial, solution to a problem. If conflict exists between two people, use classic interpersonal peacemaking techniques to help them resolve destructive differences (Walton 1969). If the conflict is between two groups of people, use team-building techniques to mediate the dispute and build and to build stronger team cohesion. Keeping People on Track Instructional designers should be able to “keep an individual or group on track so that the interaction returns quickly to its purpose and the

330

Mastering the Instructional Design Process

individual or group does not feel slighted” (Foshay, Silber, and Westgaard 1986, 99). They should be able to enact this performance when “an individual or a group wanders from the purpose of the interaction” (99). Keeping people on track means achieving desired results from interactions with others while being spared distractions of peripheral interest or concern. Why Do People Lose Track of Purpose? Any interaction between people can lose focus. Individuals are driven by different wants, needs, and goals. They often have different priorities. A meeting called to address instructional needs can turn into a platform for a handful of vocal participants to launch into a tirade against the organization’s selection, promotion, pay, or retirement practices. Likewise, an individual being interviewed about training needs may offer advice about what or who to believe in the organization. These sometimes frustrating (and sometimes amusing) mismatches between the goals and outcomes of an interaction occur because some issues weigh more heavily than others on the minds of participants. Without exercising control and exerting influence over interactions with others, instructional designers may find themselves wasting valuable time and effort or struggling to establish priorities among a myriad of competing interests. How Should People Be Guided Back on Track? Instructional designers may use several approaches in exercising control over interactions. But first they should clarify in their own minds what results they seek from a meeting, interview, or discussion. When an interaction will be lengthy and formalized, as is often the case with meetings or interviews, instructional designers should prepare an agenda or list of questions in advance. They should then send it to the participants as a place to start discussion and as a control mechanism in case others turn to tangential issues. Second, instructional designers should restate the purpose of the meeting or discussion when others wander off the topic. In one meeting a participant wanted to discuss the organization’s pay practices rather than employees’ instructional needs. (The latter topic was the reason for the meeting.) The instructional designer noted, “What you have said is most interesting and possibly true. However, I am neither qualified to judge nor knowledgeable enough on the subject to respond. Could we turn back for now to the

Managing Partnerships and Collaborative Relationships

331

subject of instructional needs?” That remark brought a prompt apology from the wanderer and renewed attention to the subject of the meeting from other participants. Third, if all else fails, an instructional designer should look for an opportunity to regain control of the meeting by directly asking the individual if it is okay to return to the topic at hand. A question such as this can also be directed more openly to the full group. This tactic can help the wanderer to realize they have strayed from the topic. While this may seem awkward and potentially embarrassing, it can help to get a discussion back on track. Using a question can feel less threatening than calling out the behavior directly and the discussion can be guided back to the subject. Fourth, use periodic feedback to keep people on track. End each meeting or discussion by asking participants, “How well did we stay on the subject of the meeting?” and “How well did we interact as a group?” This is an approach borrowed from process consultation, described by Edgar Schein (1969, 9) as “a set of activities … which helps the client to perceive, understand, and act upon process events which occur in the client’s environment.”

Securing Commitment Instructional designers should be able to “obtain commitment from an individual or group so that the commitment facilitates [the project’s] goals, both parties feel the commitment is binding, both parties are willing to follow through on it, and both parties feel there is value in it” (Foshay Silber, and Westgaard 1986, 100). Obtaining commitment means that people support the instructional design project. They will provide information, resources, to ensure that the project’s goals are achieved. Why Is Commitment Important? Instructional designers, by the work they perform, must work with—and often through—others. If they cannot secure cooperation and commitment, they will probably waste time, effort, and organizational resources as they analyze human performance problems, conduct needs assessment, and carry out other steps in the instructional design process. Hence, commitment from key decision makers and other affected groups and individuals is essential to project success.

332

Mastering the Instructional Design Process

How Is Commitment Obtained? Obtain and maintain commitment to an instructional design project by practicing empathy about the project, communicating with individuals or groups affected by the project, and encouraging participation in the project by those affected by it. These are also three key methods for managing change according to Kirkpatrick (1985). Practicing empathy means looking at a project, problem, or issue from another person’s viewpoint. It is an appreciation of the viewpoints and feelings of others. To practice empathy, instructional designers must first have information about the individual or group with whom they must interact. For instance, in preparing to deal with individuals, instructional designers may find it useful to learn about their education, experience, outside hobbies and activities, and other issues. Instructional designers might also find it useful to know what others think about instructional design projects, the present project specifically, the organization, their prospects within the organization, and any other matters that could affect their support of the project. In this way, it is possible to bring out hidden agendas and address individual concerns at the project’s outset. Communicating leads to understanding. To obtain and maintain commitment to an instructional design project, identify who will be affected by it, select timing to communicate about it, pick appropriate methods of communicating, and establish methods by which to obtain feedback (Kirkpatrick, 1985). Encouraging participation is defined as “getting involvement from those concerned with and affected by change” (Kirkpatrick 1985, 133). It is the third and final key to obtaining and maintaining commitment to an instructional design project. Since the late 1940s, numerous authors have emphasized the crucial importance of participative decision making in planning, implementing, and sustaining change (Coch and French 1948; Likert 1967; Marrow 1972; Rothwell 2015; Rothwell, Stavros, and Sullivan in press). Subsequent research has even demonstrated that participation is of critical importance in instructional design efforts in particular. Corporate planners and instructional designers feel that strategic business planning activities, typically carried out by top managers only, are more open to participative decision making than most instructional design efforts. To obtain commitment, instructional designers should begin by identifying those affected by the instructional design process. They should then use the following techniques to encourage participation in this process

Managing Partnerships and Collaborative Relationships

333

(Kirkpatrick 1985, 144): (1) ask for input before and during each step of the project; (2) consider and evaluate the input received; (3) give credit to those who contributed useful ideas; (4) thank those who contributed ideas not used, and explain why they were not used. These approaches help to increase the sense of ownership and commitment and promote adoption, change, and impact. It also helps to set the stage for future positive interactions. Participation may be solicited during each step of the instructional design process. There are good reasons for instructional designers to encourage participation by others, even after instruction has been designed. For example, different groups may be involved, and members of those groups may have their own ideas about present and future instruction.

Conducting Project Reviews A sound instructional design practice is conducting project reviews, sometimes known as after-action reviews or project debriefs. Bringing together key stakeholders to discuss what went well, what did not go so well, what was learned, what could be done next time, and other reflective questions can be a powerful way to capture key insights and improvement opportunities for future projects. The technical or project aspects of the work are not the only areas of focus of the review. In addition, the interpersonal dynamics and relationships should be reflected upon so that valuable insights can be gleaned and used going forward. This can rekindle negative emotions caused by conflict or friction, but it can also create a source of humor, as it is often easier to laugh at tense situations in hindsight. This ability to process interpersonal dynamics and find insights, emotions, and even humor can lead to powerful bonding between team members that otherwise might be lost or reduced if not discussed in a project review session.

CHAPTER NINETEEN

PLANNING AND MANAGING INSTRUCTIONAL DESIGN PROJECTS

I

nstructional design projects can grow in complexity. Simple ones may involve only a few people; complicated ones may involve teams of many people to oversee instructional design, content graphic design, and delivery systems. It is therefore important to develop a plan as a basis for monitoring the progress and scope of each project. Project planning has never been more important. This is especially true in large-scale multimedia projects where scope creep can drive up the costs and lead to long delays if projects are not artfully managed. That is most true when many experts must be coordinated in an instructional design project team (Beer 2000; Conrad and Training Links 2000; Driscoll 1998). In this chapter, we will describe how to develop a project management plan for an instructional design project, beginning with a brief discussion about the background of project management and planning. We will then describe key issues to consider when planning and monitoring projects. We will conclude the chapter with a few words about judging and justifying project plans. According to The Standards, the ability to “plan and manage instructional design projects is an advanced competency and it includes six performance statements, three are advanced and three are managerial” (Koszalka, Russ-Eft, and Reiser 2013, 66). The performance statements indicate that to plan and manage instructional design projects, 334

Planning and Managing Instructional Design Projects

335

instructional designers should: “(a) establish project scope and goals, (b) write proposals for instructional design projects, (c) use a variety of planning and management tools for instructional design projects, (d) allocate resources to support the project plan, (e) manage multiple priorities to maintain project timeline, and (f) identify and resolve project issues” (Koszalka, Russ-Eft, and Reiser 2013, 66).

Unique Challenges Posed by Project Management Project management poses unique challenges unlike those encountered in traditional line and staff management organizations. First, project managers are selected on the basis of their ability to grapple with a temporary problem or complete a unique work assignment. Team members may not have worked with each other before and may never work with each other again. The team leader must be skilled in facilitating group dynamics and team building, helping members of the group proceed quickly through the forming and storming stages in which all groups progress. Second, project managers typically lack the long-term authority over people wielded by supervisors in line and staff organizations. They are only temporary managers. Hence, project managers must be very skilled in negotiating with people and influencing them. Third, project managers exercise greater control and enjoy greater flexibility over their work assignments than most traditional managers do. As the workload necessitates, they can add or subtract team members, sharing expertise with other project managers. That is usually difficult in line and staff organizations. Project planning and managing poses its own challenges and frustrations. Its unique strengths make it well-suited to the demands of instructional design work, in which it is usually necessary to pair up experts in instructional design with experts in subject and instructional media production. Projects can spin out of control, however, for many reasons. It is not always the fault of a bad plan. Consider the following vignette described by a respondent to Rothwell’s (2003) survey, undertaken for the third edition of this book: In 2001, I was involved in developing a CD for supervisory training. I was one of an eclectic team of instructional systems specialists working on this project. The skills varied, and many of the areas for

336

Mastering the Instructional Design Process

training were apt to change significantly in the short term. On top of this a test was developed. No real analysis was done, [and] no gaps were identified beyond anecdotes. Learning objectives were developed, but they were not adequate for writing test questions over. The stakeholders for the various topics were not available or support materials such as policies and procedures had yet to be developed. The program was to be passed on to the field and updated as needed. We subsequently needed updates but had no funds for updating the CD and for reproduction costs. Poor quality of content and poor planning led to the program being passed on in a questionable condition.

This passage underscores real-world dilemmas that instructional designers face in doing project work. Establishing Project Scope and Goals It is critical at the outset of any project to clarify its scope and goals. Failure to do that will lead to projects that spiral out of control and disappoint instructional designers, managers, and learners. The question is: how to do that? There is no simple answer to that question. But seasoned instructional designers have learned lessons about how to do that based on experience. First, get all or most stakeholders in the room at the same time. If an instructional project is to cover an entire organization, then representatives of key departments should be present. If an instructional project is to include customers or clients, they, too, should be represented. One trick of the trade is to establish a governance structure such as an advisory board, composed of representatives from different stakeholder groups. If that is not possible, then conduct interviews or focus groups with representatives from those stakeholder groups. Second, make sure that the problems are clear and, if possible, measurable. As a simple example, suppose that managers feel that an organization’s customer service needs improvement. Find out how they measure that service and what behaviors they associate with excellent customer service. Project goals are somewhat akin to performance goals. They should be clear, action-focused, and measurable. Another way to think of them is the well-known SMART goals. SMART goals are specific, measurable, attainable, relevant, and time sensitive.

Planning and Managing Instructional Design Projects

337

Finally think of the project with the end results always in mind. Try to encourage stakeholders to clarify the outcomes they envision upon project completion. Try to create a shared vision of what project success will look like. Then work toward achieving it. Writing Proposals for Instructional Design Projects An instructional design proposal is like any other proposal. Most proposals have predictable parts. They include: • A statement of the problem: What is the problem to be solved? • A statement of the solution: What is the recommended solution? What is the justification for that solution? • A list of action steps: How will the solution be implemented? • A timeline: Over what timespan will the action steps be implemented? • A budget: How much will it cost over the timeline? • A staffing list: Who will work on the project, and what essential qualifications do the staff members bring to the project that will be important in completing the project? Tricks of the Trade in Writing Proposals Experienced instructional designers learn tricks of the trade when preparing project proposals. One is to focus questioning of clients around the questions listed in the section above. There is no more certain sign of novices than that they must go back to the client after initial meetings to gather additional information to write the proposal. An experienced instructional designer knows better and focuses questions on the various aspects in the preceding list to formulate a quick yet effective proposal back to the client. (Failure to respond with a proposal on a timely basis may mean losing client support—or sends a signal that the project is not a priority.) Other issues should be considered when writing proposals. One issue is that without initial data collection, it is difficult to know if the client has identified the root cause of a problem or is asking for treatments on mere symptoms only. Gathering information from people is the best way to avoid that problem, but it is difficult to do if nobody has funded (or allocated a budget for) a project. One idea is to gather data using fast, low-cost electronic methods such as e-mail prior to initial proposal submission but then gather better data later. Another idea is to write a general proposal that merely reflects back to the clients what they asked for but then bring key

338

Mastering the Instructional Design Process

stakeholders together immediately upon proposal acceptance to calibrate a better work plan. A second issue in proposal writing is to develop a budget that illustrates not just the project cost but also the likely returns to the organization. Do not wait until the end of a project to calculate return on investment; rather, gather information about the cost of the problem and then compare the benefits of solving the problem to project costs. In that way, clients feel—and properly so—that they are making an investment and will get a return rather than just spending money. Using Planning and Management Tools for Instructional Design Projects Many tools can assist instructional designers when they undertake instructional design projects. The most obvious tool is to use project management software to help to manage projects. Perhaps best known is Microsoft Project.TM It is a software program that makes it easier to track projects. However, many free alternatives to this well-known software program exist (see, for instance, www.maketecheasier.com/5-best-free-alternatives-to-microsoft-project/). But other tools are possible. A sample list includes: • • • •

Collaboration software Dashboards Scorecards Role and responsibility charting

Even software tools as diverse as Skype, Zoom, Yammer, and many others may have important applications for instructional design projects. Such tools enable virtual meetings and messaging among instructional designers.

Allocating Resources to Support the Project Plan Instructional design projects require more than project plans. They also require methods by which to foster daily, weekly, and ongoing collaboration and information sharing. Meetings are an essential part of that process. But, truth be told, challenges in allocating resources are not unique to instructional design and are a reality when managing any project type.

Planning and Managing Instructional Design Projects

339

Often, decision making about allocating resources comes down to questions of priorities. What are really the most important issues to ensure project completion? Which team members, if there is a team, have the greatest need for the resources? The answers to these questions must be provided by team leaders, or else they can ask the team for input when priorities (especially the need for resources) collide. Establishing and Using Methods to Reallocate Funds When developing a project plan, establish ways to redistribute funds, within existing project constraints, to manage differences between planned and actual project performance to meet project obligations. For instance, match the project budget to deadlines. If deadlines are not reached, or are reached sooner than expected, be prepared to compensate by making adjustments to the budget. Establish a regular schedule to review the budget and expenses compared to project deadlines. Find out, too, what procedures are used in the organization to justify budget variances, differences between planned and actual expenditures. Planning and Monitoring Equipment and Facility Requirements Consider equipment and facility requirements when preparing a plan for an instructional design project. Each project requires specialized planning for equipment and facilities. For instance, during a performance analysis, members of an instructional design team will typically need workspace and equipment. Although there is a temptation for the management of a client organization to supply whatever spare space and equipment may already be available, the project leader should see that necessary resources and facilities are requested in advance and are available when needed. To plan equipment and facilities, begin with a master scheduling and control chart for the project. For each activity on the chart, estimate equipment and facility needs. Consider what will be needed, how much will be needed, and when it will be needed. Then allow time for the equipment and facility requests to be reviewed, approved, and acted on. Establish a sign-up system for allocating equipment and facilities, since they may face conflicting demands. Make sure that the project leader resolves conflicting demands as they arise. Monitor equipment and facilities against the master scheduling and control chart. Each time a piece of equipment or a facility is used, ask instructional designers to track it. Then review equipment and facility use periodically.

340

Mastering the Instructional Design Process

Managing Multiple Priorities and Identifying/Resolving Project Issues Conflicts frequently arise during instructional design projects. One reason is there are often multiple priorities. To address these conflicts and manage multiple priorities, consult the initial project goals. If time is of the essence, the team leader should decide. If the conflict can wait for a team meeting or for e-mail responses, put the issue to a vote by the team.

PART SIX

PREDICTING THE FUTURE OF INSTRUCTIONAL DESIGN

Part VI: Predicting the Future of Instructional Design

Part I: Overview

Part V: Managing Instructional Design

Part II: Planning and Analyzing for Instructional Design

Part IV: Evaluating and Implementing Instructional Design

Part III: Designing and Developing for Instructional Design

CHAPTER TWENTY

PREDICTING THE FUTURE OF LEARNING AND INSTRUCTIONAL DESIGN

T

hroughout this book, we cover both the foundational and advanced skills and knowledge associated with becoming an effective instructional design practitioner today. But what about tomorrow and into the future? Numerous big-picture macro trends—such as globalization, demographic changes, economic, geopolitical, and technological advances—influence numerous dimensions and dynamics of culture, organizations, and life as we know it. In this final chapter we discuss the implications of such influences on learning, performance, and instructional design in organizations. We introduce a variety of innovations, emerging practices, and issues important to the future. Knowledge of these trends will enable instructional designers to not only be prepared for the future, but to help lead their organizations into the future.

Technological Advances to Improve Delivery and Design Organizations are continually adapting various technologies of all types. Sometimes the application of the technology is more general and used for purposes other than learning design or delivery, such as a video-conferencing platform. Other technologies are acquired for usage 343

344

Mastering the Instructional Design Process

in the design or delivery of learning interventions. An authoring tool is one such example. As new innovations, often accelerated or based on technologies, in learning and education emerge, and there can be a stampede of interest and enthusiasm which emerges overnight. Sometimes these ideas flame out before they are even launched, sometimes they are adapted quickly and then fizzle, and in some cases there is sustainability and adoption over the long haul. Gartner has developed a graphical image, known as the “hype cycle,” which depicts these dynamics, specifically applied to technology (see Figure 20.1). Many emerging patterns, trends, and innovations described can be viewed through the lens of the hype cycle as a way to project how they may play out. Virtual Delivery and Scale The wider usage of virtual delivery began in the post-9/11 era and accelerated during the financial crisis of the late 2000s in response to both aversion toward travel and budget cuts or restrictions. These forays into

VISIBILITY

FIGURE 20.1. Gartner’s Hype Cycle

PEAK OF INFLATED EXPECTATIONS

TECHNOLOGY TRIGGER

PLATEAU OF PRODUCTIVITY

SLOPE OF TROUGH ENLIGHTENMENT OF DISILLUSIONMENT

MATURITY Source: Gartner’s Hype Cycle. Used by permission from the website: www.gartner.com/ technology/research/methodologies/hype-cycle.jsp.

Predicting the Future of Learning and Instructional Design

345

virtual delivery helped designers and learners gain both comfort and proficiency with this means of learning. Today and into the future, aided by improvements in the technology itself, virtual delivery will continue to play an important role in scaling solutions and reaching a global and dispersed audience, while maintaining quality and effectiveness that is on par or perhaps even greater than an in-person learning experience. Simulations and Virtual Reality Simulations attempt to replicate real-life situations so learners can examine the various dynamics at play, assess options, and take actions to engage via technology in a simulated environment. Learners can practice and receive feedback on their performance as a way to improve their knowledge and skills in a safe environment prior to beginning actual on-the-job application and performance. The military and airline industries were two early forerunners of using simulation technologies to mirror combat situations and flight experiences respectively. While not an exact replication, the intent is for the simulation to get as near to reality as possible, narrowing the separation between the simulation and real life so that when the time for actual performance comes, there’s a level of proficiency that has already been acquired. The same is true of virtual realities, where environments or worlds are created that enable one or multiple people to interact with elements of the environment and with one another. Similar to simulations, virtual realities may attempt to replicate real life. Some virtual realities are limited to one person or a few people interacting with the environment while others may engage millions of people simultaneously. Massive player virtual reality games such as Minecraft and World of Warcraft are examples. A virtual reality example from the business world is created by an organization to replicate the recruiting and selection process by way of a virtual job fair. In such an environment, prospective employees can enter different rooms to learn about the organizations, ask questions, and even engage in interviews with a recruiter. Another example is a virtual reality created for new hire orientation and onboarding. One barrier associated with using these sorts of technologies has been high up-front costs to build a customized solution. If simulations involve physical equipment that goes beyond the computer-based technology only, as with a flight simulator, costs can soar into the millions of dollars. A single flight simulator may cost millions. As with any purchasing decision, the investment must be weighed against the potential benefits and costs to determine whether to proceed. Standard simulations and virtual realities

346

Mastering the Instructional Design Process

that are more off-the-shelf, with or without some ability to tailor and customize, are becoming more common, driving down the up-front expense while retaining many of the benefits. There are nearly limitless applications of simulation, virtual reality, and game-based learning approaches. As technology becomes more sophisticated, costs decrease, and the user experience grows increasingly more realistic and true-to-life, these approaches to learning and performance improvement will grow increasingly commonplace and move into more everyday life and work situations that can be simulated before trying. Ubiquitous Computing, Mobile Learning, and Wearable Technology Ubiquitous computing refers to the computer science concept of providing computing everywhere. Ubiquitous computing untethers the person from the traditional desktop computer workstation and can tap in to a variety of devices, formats, and physical locations. Mobile devices, such as laptops, cellular phones, tablets, and smartphones, have been used to both deploy and access learning wherever a connection and enough bandwidth is available for several years. Similar to how the cell phone transformed the telephone dynamic from one of place (location of the hard wired phone) to person (location independent), mobile devices have created this same opportunity for learning. Learners can receive high quality content just in time to learn a skill or perform a task whether they are in a car, in a plane, on a ship, on foot, on the shop floor, or sitting on the living room sofa or back patio. The opportunity to learn and develop can now happen anytime and anywhere. Yet sometimes people still think first of the classroom as the primary place for learning. Instructional design professionals must not only become proficient in designing instruction for mobile devices, they must help change the paradigms customers may have regarding where, when, and how learning can happen. While mobile devices for learning will continue to expand into the future, the next generation of ubiquitous computing involves wearable technologies, which is still relatively new from a widespread commercial standpoint and still emerging. The primary use of wearable technologies seems to be for entertainment, such as snowboarders wearing a GoPro camera, and everyday life applications, such as fitness tracking devices worn by athletes and health-conscious people. According to ABI Research, “Wearable computing devices are projected to explode in popularity over the next year and with a wave of new gadgets set to hit the consumer market, could soon become the norm for most people within five years.” ABI

Predicting the Future of Learning and Instructional Design

347

Research forecasts the wearable computing device market will grow to 485 million annual device shipments by 2018 (ABI Research, 2013). Just as with mobile devices, the learning and educational uses of such technologies lagged other popular culture-oriented applications. Now, learning and instructional design practitioners are running to keep up with those who are outside of the profession and building learning-oriented games and applications. Will this be the case with wearable technologies as well? Will learning professionals fail to see opportunities to use these emerging technologies to enable learning and support improved performance or will they be at the forefront? Lipman (2014) suggests that “71 percent of 16- to 24-year-olds want wearable tech” so if that prediction is even partially accurate, the possibilities and potential are enormous (Lipman 2014). Imagine, for a moment, that learners had wearable devices able to monitor physical functions like heart rate, breathing, and mental and emotional processes in the brain. Perhaps this information could be sent to the facilitators as a way to know when learning was being maximized and when it was not occurring. What if such indicators were added to our evaluation frameworks to track physiological responses and changes in neural circuitry, or mental processing indicative of learning, or lack thereof? Imagine a facilitator’s or instructional designer’s evaluation being based on the measureable changes in brainwave and cognitive functioning. Knowing when learner brain waves are not fully engaged and taking action to heighten attention and awareness could be useful tools for facilitators. Recent advances have made the ability to monitor and map brain activity associated with various stimulations a reality. What might be the application of such technologies on learning? Gamification Gamification refers to the use of gaming concepts and techniques to achieve formal and informal learning and performance outcomes. It has been used widely by organizations for marketing and customer experience and retention purposes (Van Grove 2011). More recently, gamification applies to learning and education in organizations and that trend will probably continue. Gamification attempts to tap into people’s interest in and desire for accomplishment, competition, mastery, knowledge or skill acquisition, power and status, and social interaction and camaraderie. Simple gamification involves awarding points to learners for completing or making progress on a task or through gaining new knowledge

348

Mastering the Instructional Design Process

or skill. In single-player gamification, rewards, points, or badges may be earned for achieving certain levels or thresholds or for task completion. In multiplayer scenarios, the diverging ideas of both collaboration (working with others) and competition (competing against other players) may be introduced, and one’s status may be visible to others—often through a leader board, similar to a leader board in professional golf. Competition may have downsides and unintended consequences such as cheating, unhealthy competition, and fairness issues for those who may be less proficient or interested in this activity. Various firms, initiatives, and organizations are attempting to make education, training, and organizational learning more engaging, interesting, and appealing to younger generations well versed in entertainment type gaming through the design and use of gamification. Bunchball is the name of an organization that specializes in gamification products and services (www.bunchball.com). Quest to Learn (Davidson 2009) is an example of gamification integrated by teachers and game and instructional designers into a public school system curriculum. Khan Academy (see www.khanacademy.org), launched in 2006, has grown in popularity as an educational tool that uses short educational videos on a variety of subjects for children and adults at no cost. In 2014, TrainingIndustry.com published its “first annual list designed to help buy-side organizations in their searches for the right gamification training partners” (Training Industry 2014). According to the website, the criteria used in determining the Top 20 gamification companies included features and capabilities, company size and growth potential, quality and number of clients/users, geographic reach, and awards, recognition, and competitive differentiation. Networked Learning through Social and Virtual Collaboration (Peer, Expert) The roots of networked learning can be traced to the early theory of social learning (Bandura 1963), which Bandura proposed in the 1960s and advanced in the 1970s (Bandura 1971). Social learning suggests that learning has a social component and occurs through direct observation or instruction of others. There are internal cognitive processes happening as well to make sense of that which is being observed by the learner. Expanded to a work setting, social learning can be connectivism and it brings together learning and work experiences (Griffiths and Guile 2003). Today and into the future, connectivism is considered by some to be a learning theory for the digital age (Siemens 2005).

Predicting the Future of Learning and Instructional Design

349

In organizations today, enabled by prolific technologies available in both the private and public domains, virtual collaboration has become more and more prevalent as a way of working and learning. It can occur across time zones and geographies, organizations, and hierarchical levels, formally and informally. Virtual collaboration often involves teams interacting, working, and learning together, but may also involve individuals who are not members of a team doing the same. Once more common in global organizations, remote working and virtual teaming has become more pervasive in many organizations, both global and domestic. Reaching out to seek help on a problem, supporting peers in other locations, and providing expertise to others can all occur efficiently on a massive scale through technology. Looking into the future, this trend of networked learning and virtual collaboration will continue but will expand further beyond the walls of the organization. Imagine a future where customers, suppliers, partners, colleagues, and even competitors, regulators, and attorneys all engage each other in the learning process in productive and meaningful exchanges that mutually benefit multiple parties. What a world that would be, indeed! A recent trend in the past 10 years that we expect to continue and to find its way from the university environment or academia into organizations is the emergence of Massive Open Online Courses, commonly called “MOOCs.” MOOCs are free online courses in which large numbers of participants can engage simultaneously from anywhere in the world. College and university professors, subject experts, and thought leaders typically partner with instructional design professionals, graphic/media specialists, technical experts, and others to create, organize, and post content such as videotaped lectures, course content, articles, presentations, syllabi, questions, and problems. So that it’s not a static one-way learning process, many MOOCs also include elements that foster interaction and social learning—among learners, professors, and others (Lewin 2013). MOOCs can be in the hype cycle described earlier with writer Dennis Yang wondering “Are We MOOC’d Out?” (Yang 2013). The range of topics and courses available is large and on the increase, some featuring world-class faculty from some of the most prestigious universities and institutions in the world, which bolster their credibility. Stanford University’s Stanford Online, UPEx, edX, iversity, FutureLearn, and Khan Academy are all examples of nonprofit MOOCs whereas Coursera, OpenLearn, and Udacity are commercial providers. Both nonprofit and commercial providers typically have well-known universities associated with them.

350

Mastering the Instructional Design Process

In terms of the future, as more content is added to MOOCs, organizations will integrate this readily available courseware into their internal offerings, expanding access and bringing high quality learning to bear on internal individual and organizational development needs. Organizations will also replicate the concept of MOOCs within the walls of the organization. Especially in large, global organizations, some with thousands and hundreds of thousands of people, internal and external faculty can be sourced to create courses and content that is relevant and customized to the needs of each organization. Once created, it can be provided on demand to those who wish to participate or those who are asked to participate to build a skill or competence desired by the organization. Mass Customization of Learning The industrial revolution occurred nearly 150 to 200 years ago and at the time dramatically transformed the lives of people in the United States and throughout the world. One aspect of society influenced greatly by this era was education of children and training of the workforces used to fuel greater and greater levels of production and industry. Nearly two centuries have passed and we’ve been in the Information Age, also referred to as the Digital Age or Computer Age, for many decades with advances emerging more rapidly and more prolifically. As with the Industrial Era, the Information Age has affected countless aspects of daily life for people worldwide and in all walks of life. Yet, many would argue that both education and organizational learning functions still cling to outdated practices designed to meet the needs that existed during the Industrial Era and have been laggards in leveraging the technology, information, and new learning modes successfully adopted in other arenas, from music and entertainment to shopping and mass media. In a not-so-subtle jab at one of the key principles of the Industrial Revolution, the notion of mass production, Schwahn and McGarvey (2012) consider the mass customization of learning “inevitable” in their book, titled Inevitable: Mass Customized Learning. Mass customization of learning means that the learning needs of each individual are identified such that unique instruction is provided when, where, and how the person prefers. An excellent parallel can be seen in how iTunes revolutionized the music industry. Instead of the one-size-fits-all approach previously provided by record producers, where buyers needed to go to the music store and purchase full albums by their favorite artists, iTunes enables the music purchaser to select only

Predicting the Future of Learning and Instructional Design

351

what is of interest and can download it and listen to it from virtually anywhere. It turned the music world on its head and put the person with their mobile device, and his or her needs, at the center of it all. This provides a vision for what learning could be. Taking that a step further, not only will learning resources be available and targeted at the unique needs of the learner, learning will be served up based on the preferences, behaviors, and needs of the learner using predictive analytics and technologies similar to how Google and Amazon suggest purchases based on consumer searching and buying patterns. In this future, just-in-time and just enough learning will enable the right learning to be available in the right form, when it’s needed, how it’s preferred by the learner, and wherever the learner is at the point of need, interest, or desire. Outsourcing Instructional Design Outsourcing instructional design involves contracting with an external organization that specializes in all or various aspects of the ID process. It allows an organization to pay for services if and when needed, tap into deep expertise in ID, and manage expenses in variable rather than fixed ways when ID resources are full-time employees of the organization. A disadvantage of outsourcing is that the external firm may be less familiar with important cultural, political, or organizational dynamics or issues. This drawback can be reduced when a long-term relationship is established between the outsourced provider and the client organization. A mixed model is when there’s a small internal core of instructional design capability, with the ability to tap into external resources as needed. Often, the decision to outsource is based on cost reduction, but other factors may come into play, too, such as flexibility, other priorities, the organization’s position regarding outsourcing, core competence, and others. Organizations that provide outsourced instructional design services range from independent consultants to large corporations with deep expertise, global reach, and state-of-the art technological capabilities. While broader learning services may be contracted for on a fixed contract basis, the nature of instructional design services lends itself to a variable or pay-as-you-go model that is more project oriented. Global Sourcing of Content and Design A trend in the outsourcing arena likely to continue into the future relates to global sourcing of instructional design services. Globalization, technology

352

Mastering the Instructional Design Process

capabilities, labor costs, and growing ID expertise enables instructional design work to be performed offshore. India and countries like Malaysia are being tapped as low-cost sources for ID support and services. As capabilities are developed across the globe and borders become more porous, aided by technology advances, the sourcing, procurement, and access to instructional design resources will become more and more accessible. Open-Sourced Design and Development The term open source has referred to the practice in the software design and development domain in which computer programming code used in various software platforms and applications is provided to the public. As individuals with varied backgrounds and expertise access the codes, improvements are made. Open source decentralizes and democratizes the software development process, compared to a centralized model where ownership, protection, and access is tightly controlled by the owner and those who have restricted access. Open source applied to learning is a practice that could easily find its way into the instructional design and content development arena. Similar to MOOCs, discussed earlier, in which university faculty are providing course material and free access to anyone, instructional design of content of all sorts could also be provided via the Internet. Imagine someone who requests support for an instructional design project on which they are working. Experts and practitioners could create and provide material and content and provide virtual accessibility. Instructional design experts could collaborate and contribute ideas and support to make such content instructionally sound and high quality. Depending on the purpose and needs being addressed, instructional designers may offer their services free, as is true with open source software. Or perhaps the party interested in the final product may offer payment or incentive to those who were involved in the design and development process. Intellectual Property and Copyright Issues In an open-source instructional design scenario, as with many traditional design projects, intellectual property ownership, copyright, quality, and validity become important and sometimes difficult to ensure. The more open ID processes become, the easier it may be for people to knowingly or unknowingly violate copyright issues, not properly cite sources, not

Predicting the Future of Learning and Instructional Design

353

seek permission to use, and not take the time and effort to validate and verify sources and ensure that the work being done is of the utmost quality and integrity. Safeguards must be established, awareness of issues must be raised, and actions must be taken so that high quality content is produced but in a way that is legally defensible and beyond reproach. As new ground is charted amid the onslaught of new knowledge creation, warp speed of change, and technology advances, expect that these issues will continue to grow in complexity and importance. Rapid Deployment from Need to Solution In an age of instant gratification, information access, and blazingly fast technologies, organizations and individuals expect quick answers to their problems or the goals they are trying to achieve. They not only grow impatient when there are delays, but the problems themselves may continue to morph, expand, and spawn other problems if not addressed expediently. A trend that will continue regarding instructional problems is the need for rapid design and development. Needs assessments must quickly detect, verify, and scope problems. Once design and development begins, it must happen with both speed and quality. Designing interventions rapidly does not mean haphazardly or without sound ID practices—high standards must be maintained, but they cannot take forever to achieve. Once designed, solutions must be deployed quickly and smoothly to ensure adoption and impact. Advancements in instructional design software help to automate and reduce cycle times of the design and development process. Sophisticated delivery platforms such as virtual collaboration software designed for learning, high-fidelity video conferencing, and remote telepresence devices open up new avenues to achieve scale of solutions in less time. Whether directly developed for learning or not, these capabilities will continue to improve as new technologies emerge and existing ones become refined, both of which will enable more rapid development and deployment. Another trend that will continue is improved quality of animation, video, and instructional design authoring capability. Advances from the motion picture, mobile applications, and video-gaming industries will lead to advances in the learning and development and ID professions. The ability to embed realistic, high-fidelity, multimedia experiences into a learning resources will be easier and more readily available. Instructional designers,

354

Mastering the Instructional Design Process

with relatively low-cost equipment and authoring software tools, can create experiences that engage participants and accelerate learning. Just as younger generation, or millennial, workers expect organizations to provide cutting-edge mobile and computing capability, the expectation that online learning will be on par in quality and experience with video and computer gaming will continue to grow. Organizations that fail to provide high-quality and highly relevant content insisted upon by learners will face engagement and perhaps even retention issues as astute learners seek such opportunities elsewhere. Proliferation of Video-Based Learning With the emergence of video sharing websites, such as YouTube, subsequently purchased by Google, the ability to both upload and access video-based content has become pervasive. A multitude of personal interest learning-oriented videos, from guitar instruction to cooking to gardening to home remodeling, are available along with a countless number of humorous, inspirational and even disgusting videos that have been posted by amateurs and professionals. In addition, numerous workplace or professional videos are also available on various subjects, such as leadership and management, project management, career development, and many others. MOOCs, mentioned earlier, often use videos as the primary delivery method, often surrounded by other content, resources, or tools. While the capability and quality of video-based production continues to increase, and while costs simultaneously decrease, there will also continue to be a tolerance for video-based learning that is lower in quality. This is due to a surprising willingness to view, which may mean millions of views, so-called “viral” videos, compelling content that captures an audience regardless of video quality. If the material is of high interest, relevance, or engagement, viewers will overlook choppy, grainy, or inconsistent footage shot with a handheld device. We anticipate an increase in the use of video-based learning in organizations, where short instructional, informational, or learning-oriented videos can be captured by anyone in organization and posted, perhaps externally on sites like YouTube or on internally hosted delivery platforms. As with YouTube-style videos, the quality of the footage will be less important than the content, and learners will access posted material just in time to gain awareness, acquire new knowledge, build a skill, or perform a task.

Predicting the Future of Learning and Instructional Design

355

Informal Learning Instructional design is typically thought of in terms of the creation of formal learning, but what role does ID play in informal learning? Informal learning is a growing area of focus among researchers and practitioners and it will continue to receive attention primarily due to its prevalence relative to formal learning. One way to think about what may be included in the informal learning arena is what happens outside of the classroom, online, or structured learning situation to help one acquire knowledge or improve performance professionally and personally. The graph in Figure 20.2 shows the relative magnitude of informal learning compared to formal learning in a study of time to performance (Moore 1998). Doing a Google search to learn about a medical condition you may have is an example of informal learning. A conversation with a peer or coworker about a problem you’re facing is another. Being asked to play the role of project leader on a new cross-functional initiative, volunteering to serve on a nonprofit board, learning to ski, and becoming a parent are other examples, and the list goes on. Even the conversation at the proverbial water cooler could represent informal learning. A common theme of FIGURE 20.2. Time to Learning Performance over Time

75%

I Adopt and Adapt Informal Learning

20% 5%

I Can Do

Formal Learning

I Know Study by Sally Anne Moore, Digital Equipment Corporation “Time to Performance.”

Source: S. Moore, “Time-to-Learning.” Digital Equipment Corporation (1998). Used by permission. Found on website: http://en.wikipedia.org/wiki/File:Ttp_samoore.gif.

356

Mastering the Instructional Design Process

many is that all of these examples represent learning through experiences and learning through others. Informal learning is often about learning through trial and error or trial and success. The notion of self-directed learning, lifelong learning, or being a continuous learner can be synonymous with engaging in informal learning. The incredible amount of, and easy access to, information of all sorts through the Internet has expanded the ability of a self-directed learner to become proficient, and even an expert, on countless subjects. Powerful search engines return results that provide answers to questions, solutions to problems, and knowledge that a short time ago was limited to direct contact with experts or accessed only by reading a textbook or paper. For many, when they face a problem, their first response is to “Google it” or turn to many online sources such as Wikipedia or Ask.com. While questions of quality, accuracy, and currency of information found informally via the Internet must be questioned and scrutinized, it provides a just-in-time, accessible from anywhere knowledge base that the majority of people and employees turn to for learning. With both the size of informal learning and the role it plays in actual performance, what role should ID professionals play? Should the focus of our craft be on the formal world of learning where the majority of organizational investments are made? There are several opportunities to add value in both arenas and play a role in bringing these worlds together. There will be significant opportunities for ID professionals to help their organizations to formalize informal learning so learning through experiences can be more structured, more purposeful, and more rapid, through ID and expertise being applied more creatively. Another opportunity is to blend informal learning into formal instructional design efforts. This is often done through instructional activities discussed previously, such as group discussions or activities, but there likely will exist many other ways to incorporate informal learning practices and create an even richer and impactful experience. The Future Learner One way to predict what the future of learning may hold is to study and anticipate the future learner. Millennials, often called Generation Y or the Millennial Generation, are those born between the early 1980s through the mid-1990s. Many are in their twenties and thirties and therefore of working age. Traits commonly ascribed to Millennials include achievement oriented, socially connected, optimistic, technology proficient, civic

Predicting the Future of Learning and Instructional Design

357

minded, and tolerant, and also impatient, self-centered, and risk averse. They have a strong desire for openness and transparency, recognition, challenge, career advancement, balance and flexibility, collaboration, and team working. What are the implications of these and other characteristics on the world of work and learning for preferences, styles, and tendencies? How can organizations create a unique value proposition based on what’s more important to this generation as a way to attract, engage, develop, and retain those who will represent the workforce and senior leadership of organizations well into the future? At 80 million in number, they are formidable in size and organizations that don’t embrace them and change may find themselves in a talent deficit. What about the post-Millennials? They don’t even have a name that has stuck. Possibilities include Generation Z, the Net Generation, the iGeneration, or the Digital Natives. One thing is certain, as the last potential name suggests, they will be the most technologically proficient generation yet, since literally from birth they have been surrounded by technologies not even imagined by preceding generations. This generation will have the highest comfort level with new innovations as they are introduced. But how will they best learn? What will be their predominant preferences and styles? What alterations to instructional design will be required to meet this next generation of learners where they are at and where they are heading? Will instructional design as we know it today even be relevant? As this newest generation, which does not yet have a defined end time horizon, continues to emerge, learning and ID professionals must closely monitor their habits, traits, likes, and dislikes because these will be the early signals of their expectations and preferences as they enter the workforce.

The Impact of Neuroscience Research on Instructional Systems Design As we’ve noted in various sections of this book, the advances made in technology today have had a tremendous impact on learning overall and, in particular, on the field of instructional design. It’s important to note, however, that other fields of study that are making tremendous strides in their own right are also exerting an influence on instructional design. One of these fields is the study and practice of neuroscience, or how the human brain functions and, in particular, learns. While the reader will find no shortage of research articles, books, presentations, and blogs on the subject, a brief

358

Mastering the Instructional Design Process

summary of how neuroscience is changing the world of the instructional designer needs to be presented as part of this work. The authors encourage readers to explore the topic in much more depth than will be presented here. Our goal is to highlight two general topics related to neuroscience and ISD: 1. Advances made in the area of neuroscience research. 2. The impact of these advances on the process of instructional design. What is interesting to note is that some of the key principles under which instructional designers have labored for years are being substantiated by some of the neuroscience research findings. On the other hand, some of our cherished principles and practices are being called into question as we uncover more about how the brain learns. These challenges to tradition are not new, either. For example, Clark (2010) demonstrated that evidenced-based research calls into question the validity of such practices as left-brain versus right-brain learning styles and the indiscriminate use of media as a panacea for all learning solutions. Clark cautions us about relying too heavily on a single approach without the support of sound evidence. As we explore the relationship of neuroscience and ISD, we need to apply a healthy dose of skepticism to any claim that isn’t supported by sound and verifiable research. Major Advances in Neuroscience Research For years now, it was believed that the brain’s ability to learn began to deteriorate around the age of eighteen. Recent findings, however, have demonstrated that the brain remakes itself almost every day (Meacham 2015). This concept, knows as neuroplasticity, suggests that learners may well be “learning” far more, or far differently, than we are aware of as instructional designers. The billions of neurons that form the foundation of our brain’s operation constantly change their structure based on experience and environmental factors. Another major discovery in recent years is the fact that the brain and the body are not separate entities, but are interdependent, as are the concepts of logic and emotion. Damasio (1994), in his book Descartes’ Error, makes the point that thinking and feeling cannot be separated, as Western philosophy, psychology, and medicine have often tried to do. Since emotion is as important to learning as thinking and logic, we need to consider ways to use emotion, particularly in gaining our learners’ attention.

Predicting the Future of Learning and Instructional Design

359

Another replicated finding is that, in spite of what younger generations might believe, the ability to “multitask” is a fallacy. Research demonstrates that we can only focus on one conscious task at a time; more than that, and accuracy and overall performance drops off dramatically. In an attempt to represent the operation of the human brain, several models, most notably information processing models and those based on how a computer operates, have proven to be too limiting in their descriptions of how the brain functions. Based on modern neuroscience research, a parallel processing models appears to be the best fit; that is, the brain processes many items quickly and simultaneously, with memory being dynamic and ever-changing based on experience (Sousa 2011). While these findings merely highlight the advancements made by neuroscience research, we can now correlate much of what we know about how the brain learns to the practice of designing instruction. The remainder of this section highlights several areas of brain research related to learning and memory and the implications for training and instructional design professionals.

Implications for instructional designers Table 20.1 identifies some of the ISD practices that are suggested by what we’re learning about brain functioning. It is not intended to be comprehensive and continuing research in the fields of neuroscience and learning will shed new light on the subject. Some of these practices are not new while others tend to refute long-held beliefs. New findings are coming every day and the reader is encouraged to stay current with these advances. Taylor (2006) provides an interesting synopsis of the impact of neuroscience research on ISD. Her findings are summarized here: 1. For learning to be successful, it must be meaningful. Particularly with new learning, the learner is searching for how this new learning relates to what he or she already knows or has already experienced. Content may be meaningful to the instructor, but, unless it is shown to be meaningful to the learner, it likely will not be retained over time. 2. Brain research appears to support the fundamental tenets of constructivism and experiential learning. Though these were theories in the past (e.g., Kolb’s Experiential Learning theory), they seem to have been validated by today’s neuroscience research. Experimentation suggests that

360

Mastering the Instructional Design Process

TABLE 20.1. IMPLICATIONS OF NEUROSCIENCE RESEARCH FOR ISD Function

Neuroscience Finding

Implications for ISD

Continuous learning (neuroplasticity)

The brain is shaped by both its genetic code and its environment (Wolfe, 2010)

• Incorporate the learner’s environment into the design and delivery of instruction • Design instruction that (a) begins with prior experience; (b) connects new learning to past experience; (c) creates a unique experience and supports it with repetition (Sheckley & Bell, 2006)

Logic and emotion

The brain systems controlling • Learning activities that incorporate emotion are more likely to gain the emotion, feeling, reason, and learner’s attention and to embed the decision-making interact with learning into long-term memory (Wolfe, one another 2006) • When learners “like” what they’re learning, they are more attentive and their critical thinking skills are enhanced (Sousa, 2011)

Attention and short-term memory

The brain must be aroused for • Instruction must be presented from the beginning so that it attracts the attention it to pay attention and its and interest of the learner ability to hold information is • Information must be presented in “chunks” transient and short term and must be repeated or rehearsed so that (Wolfe, 2010) it can be stored in long-term memory • New information must be relevant to the learner’s experience • Rehearsal of new information can be rote (over and over) and elaborative (learner expands or elaborates on new information to enhance meaning and relevance) • We are more likely to remember information longer the more it was introduced and reinforced through emotion • Long-term memory consists • Declarative memory describes our ability to of two types: declarative store and recall information (e.g., names, and procedural dates, events) • Procedural memory does • Procedural memory reflects the ability to not use the same brain describe how something works (e.g., structures as those used reading, cooking a roast) with declarative memory • Elaborate rehearsal structures are more likely to enhance retention of newly-learned information in long-term memory, particularly when those strategies are distributed over time • The more that we process information over time, the more neural connections are made, increasing the likelihood that newly-learned information will be retained for a long time

Long-term memory

Predicting the Future of Learning and Instructional Design

361

TABLE 20.1. IMPLICATIONS OF NEUROSCIENCE RESEARCH FOR ISD cont’d. Function

Neuroscience Finding

Implications for ISD

Transfer of learning

• Information processing depends on connecting past learning with new learning • Learners must be able to see how new learning can be used in the future

• Connecting new learning with past experience (context) enhances transfer (positive transfer) • Past experience that conflicts with new learning inhibits transfer (negative transfer) • Transfer is not facilitated by rote (repetitive) learning • Research shows transfer is facilitated by (a) the context and degree of original learning; (b) similarity; (c) critical attributes and (d) association (Sousa, 2011)

learning is constructed in the mind of the learner and is enhanced when the learning is meaningful to the learner. 3. Different parts of the brain are used when exposed to veridical learning (getting the “right” answer to a question or activity) than when exposed to nonveridical learning (examining problems from more than one perspective and reflecting on the problem). Much training today focuses on veridical learning, but, since meaningful learning is constructed and retained over time, ISD must increase the use of nonveridical learning for the best long-term results. 4. New experiences are more likely to be transferred to long-term memory the more emotional the experience. Storytelling is a powerful way of eliciting emotion when introducing new learning. The merger of instructional design and neuroscience is a marriage made in heaven! The more we discover about how the brain learns and remembers, the more likely it is that the field of instructional design will alter and adapt its methodologies to accommodate the emerging world of brain research.

A Final Word “The only thing that is constant is change has been used so frequently that we need not even complete the sentence. The question we wish to raise as we conclude this chapter and book is how will the instructional design profession not only prepare for but proactively embrace and welcome the

362

Mastering the Instructional Design Process

future. Will we hang onto old ways that are no longer relevant? A “gut check” response to that question is to ask ourselves how well we’ve updated our practices and stayed relevant with the changes that have occurred in the past decade. Will we actively seek to change the practice and profession of instructional design or will we cling to what’s worked in the past and resist, tolerate, or completely ignore the need to adapt? Instructional design has played a unique role in organizations over more than half a century. If instructional design involves devising “instructional experiences, which make the acquisition of knowledge and skill more efficient, effective and appealing” (Merrill, Drake, Lacy, Pratt, and ID2 Research Group 1996), is our role broader than formal learning interventions? Even using Merrill’s definition from the mid-1990s, it leaves open the possibility to have broad reach and impact far beyond the design of formal learning experiences. This is likely where the future is heading. Will ID practitioners lag behind or lead the way? The best choice is to lead the way in this dynamic and uncertain future. We must establish a compelling “value proposition” for ourselves individually, for our workplaces, and for our profession. Doing so will ensure that we not only embrace these and numerous unknown future trends, but also stay highly relevant and ahead of the curve.

ABOUT THE AUTHORS

William J. Rothwell, PhD, SPHR, CPLP Fellow is President of Rothwell & Associates, Inc., and Rothwell & Associates, LLC (see www.rothwellandassociates.com). He is also a Professor in the Workforce Education and Development program, Department of Learning and Performance Systems, at Pennsylvania State University, University Park campus. He has authored, coauthored, edited, or coedited 300 books, book chapters, and articles—including over 90 books. Before arriving at Penn State in 1993, he had 20 years of work experience as a Training Director, in HR, and in Organization Development in government and in business. He has also worked as a consultant for more than 40 multinational corporations—including Motorola China, General Motors, Ford, and many others. In 2012, he earned ASTD’s prestigious Distinguished Contribution to Workplace Learning and Performance Award, and in 2013 ASTD honored him by naming him as a Certified Professional in Learning and Performance (CPLP) Fellow. In 2014, he was given the Asia-Pacific International Personality Brandlaureate Award (see www .thebrandlaureate.com/awards/ibp_bpa.php). Truly global in his outlook, he has conducted training in 15 nations around the globe in 2014 alone and visited many other nations virtually to offer webinars. Notably, in 2015 he visited China for his 76th time since 1996. 363

364

Mastering the Instructional Design Process

His recent books include Beyond Training and Development, 3rd ed. (HRD Press, 2015), Career Planning and Succession Management, 2nd ed. (Praeger, 2015), Organization Development Fundamentals: Managing Strategic Change (ATD Press, 2015), The Competency Toolkit, 2 vols., 2nd ed. (HRD Press, 2015), Creating Engaged Employees: It’s Worth the Investment (ATD Press, 2014), The Leader’s Daily Role in Talent Management: Maximizing Results, Engagement, and Retention (Institute for Training and Development [Malaysia], 2014), Optimizing Talent in the Federal Workforce (Management Concepts, 2014), Performance Consulting (John Wiley & Sons, 2014), the ASTD Competency Study: The Training and Development Profession Redefined (ASTD, 2013), Becoming an Effective Mentoring Leader: Proven Strategies for Building Excellence in Your Organization (McGraw-Hill, 2013), Talent Management: A Step-by-Step Action-Oriented Approach Based on Best Practice (HRD Press, 2012), the edited three-volume Encyclopedia of Human Resource Management (Wiley/Pfeiffer, 2012), Lean but Agile: Rethink Workforce Planning and Gain a True Competitive Advantage (AMACOM, 2012), Invaluable Knowledge: Securing Your Company’s Technical Expertise-Recruiting and Retaining Top Talent, Transferring Technical Knowledge, Engaging High Performers (AMACOM, 2011), Competency-Based Training Basics (ASTD Press, 2010), Effective Succession Planning: Ensuring Leadership Continuity and Building Talent from Within, 4th ed. (AMACOM, 2010), Practicing Organization Development, 3rd ed. (Pfeiffer, 2009), Basics of Adult Learning (ASTD, 2009), HR Transformation (Davies-Black, 2008), Working Longer: New Strategies for Managing, Training, and Retaining Older Employees (AMACOM, 2008), and Cases in Government Succession Planning: Action-Oriented Strategies for Public-Sector Human Capital Management, Workforce Planning, Succession Planning, and Talent Management (HRD Press, 2008). He can be reached by e-mail at [email protected] and by phone at 814-863-2581. He is located at 310B Keller Building, University Park, PA 16803. G.M. (Bud) Benscoter, PhD, is the owner of GMB Performance Group, a performance improvement consulting company. Bud has more than 25 years of experience in marketing, sales, training, organization development, and management. He has worked with numerous companies in the financial services, technology, pharmaceutical, and manufacturing sectors as well as federal government agencies. Bud has a master’s degree and PhD in instructional systems design from Penn State, where he taught for nearly 20 years in both the Instructional Systems and MBA programs. He received the Excellence in Teaching Award from the Smeal College of Management in 1997 and 2005. Bud also taught management communications at the Wharton School of Business at

About the Authors

365

the University of Pennsylvania from 2002 to 2004 and in the Fuqua School of Business at Duke University from 2006 to 2010. He was editor of the Encyclopedia of Human Resource Management, Volume 3 published by Pfeiffer in 2011 and coauthored Performance Consulting published by John Wiley & Sons in 2014. Marsha King, PhD, is the Founder and President of SkillPoint Consulting, LLC, where she is an independent consultant working with senior leaders to assess their skills and provide coaching in the areas of leadership and management. Most recently, she has coached executives within Navigant Consulting, Hanover Insurance, B/E Aerospace, T. Rowe Price, Federal Reserve Banks, Nationwide, Dale Carnegie, Gurwitch Cosmetics, CEVA Logistics, The Goodwill of Greater Washington, Sunrise Senior Living, Pepco Holdings, and AARP. Marsha worked for Capital One Financial, a Fortune 200 Financial services firm based in McLean, VA, for nearly eight years. She began as the Director of Leadership Acceleration, where she led 40 employees to provide services across the organization including succession planning, performance management, diversity, training and development, and talent management. After three years, she became a Managing Vice President of HR and led HR Consulting for Capital One’s staff groups and several other strategic businesses. Additionally, she led HR branding and enterprise-wide change management as part of her responsibilities. Prior to joining Capital One, Marsha was an Executive Coach for Development Dimensions International (DDI), a global human resource consulting firm based in Pittsburgh, PA. She worked with several Fortune 500 firms including Chase, Qwest, Nortel, Steelcase, Whirlpool, and the U.S. Postal Service to assess leadership capability and coach leaders to demonstrate stronger performance. Prior to DDI she worked with the Pennsylvania State University as a faculty coach in their Engineering Leadership Program. Marsha began her career with Worthington Industries, headquartered in Columbus, Ohio, as a human resource generalist where she specialized in training and development and performance improvement. Marsha is also a well-respected adjunct professor at Northwestern University, Georgetown University, and George Washington University, where she teaches in their Human Resource Development programs, specializing in Leadership and Executive Coaching in the United States and across Asia. Stephen B. King, PhD, is currently Vice President, Talent & Leadership Effectiveness for Allstate Insurance Company, with responsibility for Enterprise Learning, Talent, Succession Management, Performance Management, Leadership & Executive Development, and Energy for Life (a

366

Mastering the Instructional Design Process

strategic initiative to help Allstaters identify their purpose and manage their energy effectively). Steve is a graduate of the Ohio State University and Pennsylvania State University. He began his career in a variety of operations roles with Worthington Industries. Before joining Allstate, Steve held key leadership roles in leading organizations. He was the Director of Leadership & Management at a global consulting firm, Management Concepts, after which he took the role of the Chief Learning & Talent Officer with Constellation Energy in Baltimore, Maryland. In November 2008, Steve joined Allstate’s Human Resource Senior Leadership Team. Steve and his wife, Marsha, have one son and reside in Glenview, Illinois. In addition to spending time with his family, part of Steve’s personal purpose is helping children in need through his involvement in Safe Families for Children. He also enjoys fitness, gardening, and strumming his guitar.

REFERENCES

Abernathy, W. B. 2010. “A Comprehensive Performance Analysis and Improvement Method.” Performance Improvement 49 (5): 5–17. doi:10.1002/pfi.20146. ABI Research. 2013. “Wearable Computing Devices, Like Apple’s iWatch, Will Exceed 485 Million Annual Shipments by 2018.” www.abiresearch.com/press/wearablecomputing-devices-like-apples-iwatch-will. Academy for Educational Development. 2006. Introduction to Data Analysis Handbook. Allen, M. 2012. Leaving ADDIE for SAM . Alexandria, VA: ASTD Press. Ambrose, S., M. Bridges, M. DiPietro, and M. Lovett. 2010. How Learning Works: Seven Research-Based Principles for Smart Teaching . San Francisco: Jossey-Bass. Annett, J., and N. Stanton. 2001. Task Analysis. New York: Taylor & Francis. Arneson, J., W. Rothwell, and J. Naughton. 2013. ASTD Competency Study: The Training and Development Profession Revisited. Alexandria, VA: ASTD Press. Arshavskiy, M. 2013. Instructional Design for eLearning . Marina Arshavskiy (publisher). Arvey, R., and R. Faley. 1988. Fairness in Selecting Employees. 2nd ed. Reading, MA: Addison-Wesley. Ausubel, D. 1962. “A Subsumption Theory of Meaningful Verbal Learning and Retention.” Journal of General Psychology 66: 213–214. Bachman, L. 1987. “Pilot Your Program for Success.” Training and Development 41 (5): 96–97. Bader, G., and C. Rossi. 2002. Focus Groups: A Step-by-Step Guide. 3rd ed. San Diego, CA: The Bader Group. Baker, H., and P. Morgan. 1989. “Building a Professional Image: Handling Conflict.” In F. Stone (Ed.), The American Management Association Handbook of Supervisory

367

368

References

Management. (Originally printed in Supervisory Management, September 1980.) New York: AMACOM. Baldwin, T., and J. Ford. 1988. “Transfer of Training: A Review and Directions for Future Research.” Personnel Psychology 41 (1): 63–105. Bandura, A. 1963. Social Learning and Personality Development. New York: Holt, Rinehart, and Winston. Bandura, A. 1971. Social Learning Theory. New Brunswick, NJ: Transaction Publishers. Barbazette, J. 2006. Training Needs Assessment. San Francisco, CA: Pfeiffer. Barbian, J. 2002. “Blended Works: Here’s Proof.” Online Learning 6 (6): 26–31. Basarab, D. 2011. Predictive Evaluation. San Francisco, CA: Berrett-Koehler. Beckshi, P., and M. Doty. 2000. “Instructional Systems Design: A Little Bit of ADDIEtude, Please!” In The ASTD Handbook of Training Design and Delivery, edited by G. Piskurich, P. Beckshi, and B. Hall, 28–41. New York: McGraw-Hill. Beer, V. 2000. The Web Learning Fieldbook: Using the World Wide Web to Build Workplace Learning Environments. San Francisco: Pfeiffer. Bengtson, B. 1994. An Analysis of CEO Perceptions Concerning Trainer Roles in Selected Central Pennsylvania Manufacturing Firms. Unpublished doctoral dissertation. University Park, PA: Penn State University. Bentley, T. 1994. Facilitation: Providing Opportunities for Learning . London: McGraw-Hill. Bersin, J. 2004. The Blended Learning Book. San Francisco, CA: John Wiley & Sons. Berin, J. 2008. The High Impact Learning Organization: What Works® in the Management, Governance and Operations of Modern Corporate Training . Oakland, CA: Bersin & Associates, LLC. Bishop, M. J. 2014. “Reconceptualizing Instructional Message Design: Toward the Development of a New Guiding Framework.” Design in Educational Technology, 143–159. Blank, W. 1982. Handbook for Developing Competency-Based Training Programs. Englewood Cliffs, NJ: Prentice-Hall. Bozarth, Jane. 2010. Social Media for Trainers. San Francisco, CA: Pfeiffer. Brinkerhoff, R. O. 2010. The Success Case Method: Find Out Quickly What’s Working and What’s Not. San Francisco: Berrett-Koehler. Brinkerhoff, R. 2005. “The Success Case Method: A Strategic Evaluation Approach to Increasing the Value and Effect of Training.” Advances in Developing Human Resources 7 (1): 86–101. http://aetcnec.ucsf.edu/evaluation/Brinkerhoff.impactassess1.pdf. Broad, M., and J. Newstrom. 1992. Transfer of Training: Action-Packed Strategies to Ensure High Payoff from Training Investments. Reading, MA: Addison-Wesley. Campbell, D., and J. Stanley. 1966. Experimental and Quasi-Experimental Design for Research. Chicago: Rand-McNally. Carnevale, A., L. Gainer, and A. Meltzer. 1988. Workplace Basics: The Skills Employers Want. Alexandria, VA: American Society for Training and Development and Washington, DC: Employment and Training Administration, U.S. Department of Labor. Carruth, P. J., and A. K. Carruth. 2012. “Using eLearning to Build Workforce Capacity Among Business Professionals.” American Journal of Business Education 5 (6): 709–718. Chapman, B. 2010. “How Long Does It Take to Create Learning?” A Chapman Alliance Research Study, September.

References

369

Charness, N., and S. Czaja. 2006. Older Worker Training: What We Know and Don’t Know. Washington, DC: AARP. www.aarp.org/research/work/issues/2006_22_worker .html. Chase, W., and M. Chi. 1980. “Cognitive Skill: Implications for Spatial Skill in Large-Scale Environments.” In Cognition, Social Behavior, and the Environment, edited by J. Harvey, 111–136. Hillsdale, NJ: Lawrence Erlbaum. Chen, B., and T. Bryer. 2012. “Investigating Instructional Strategies for Using Social Media in Formal and Informal Learning.” The International Review of Research in Open and Distributed Learning 13 (1): 87–104. Chernick, J. 1992. “Keeping Your Pilots on Course.” Training and Development, 6 (4): 69–73. Chinien, C., and F. Boutin. 1994. “A Framework for Evaluating the Effectiveness of Instructional Materials.” Performance and Instruction 33 (3): 15–18. Clark, R. 2003. Building Expertise. Washington, DC: International Society for Performance Improvement. Clark, R. C. 2010. Evidence-Based Training Methods. Alexandria, VA: ASTD Press. Clark, R. 1999. Developing Technical Training . Washington, DC: International Society for Performance Improvement. Coch, L., and J. French Jr. 1948. “Overcoming Resistance to Change.” Human Relations (August): 512–532. Conrad, K., and Training Links. 2000. Instructional Design for Web-Based Training . Amherst, MA: Human Resource Development Press. Covey, S. R. 1989. The Seven Habits of Highly Effective People. New York: Free Press. Crabtree, S. 2007. “Getting Personal in the Workplace: Are Negative Relationships Squelching Productivity in Your Company?” The Gallup Management Journal. www .govleaders.org/gallup_article_getting_personal.htm. Crowe, M., L. Hettinger, J. Weber, and J. Johnson. 1986. Analysis of Students’ Basic Skills Performance in Selected Instructional Delivery Systems: Final Report. Columbus, OH: National Center for Research in Vocational Education, Ohio State University. Dalto, Jeffrey. 2013. Retrieved from convergencetraining.com/2013/02/tips-forwriting-instructional-and.html. Dalton, G., P. Thompson, and R. Price. 1977. “The Four Stages of Professional Careers: A New Look at Performance by Professionals.” Organizational Dynamics (Summer): 19–42. Dalton, M. 1969. “Conflict Between Staff and Line Managerial Officers.” In A Sociological Reader on Complex Organizations (2nd ed.), edited by A. Etzioni. Troy, MO: Holt, Rinehart and Winston. Damasio, A. 1994. Descartes’ Error . New York, NY: Penguin Book. Darr, G., and N. M. Seel. 2014. “Instructional Delivery Systems and Multimedia Environments.” Instructional Design. International Perspectives 2:145–181. David, M. 2002. Guide to Successful Executive Coaching . Info-Line, No. 250204. Alexandria, VA: The American Society for Training and Development. Davidson, C. 2009. “Game School Opens in New York: Quest to Learn.” February 16. www.hastac.org/blogs/cathy-davidson/game-school-opens-new-yorkquest-learn. Davis, R., Aleander, L., Yelon, S. 1974. Learning Systems Design. New York: McGraw-Hill. Debrovolny, J. L., and S. C. G. Fuentes. 2008. “Quantitative versus Qualitative Evaluation: A Tool to Decide Which to Use.” Performance Improvement 47 (4): 7–14.

370

References

Delahoussaye, M., and R. Zemke. 2001. “10 Things We Know for Sure.” Training 38 (9): 48–59. Dick, W., and L. Carey. 1990. The Systematic Design of Instruction. 3rd ed. New York: HarperCollins. Dick, W., and D. King. 1994. “Formative Evaluation in the Performance Context.” Performance and Instruction 33 (9): 3–8. Dillman, D. 1999. Mail and Internet Surveys: The Tailored Design Method. 2nd ed. Hoboken, NJ: John Wiley & Sons. Dixon, N. 1990. “The Relationship between Trainee Responses on Participant Evaluation Forms and Posttest Scores.” Human Resource Development Quarterly 1 (2): 129–137. Driscoll, M. 1998. Web-Based Training: Using Technology to Design Adult Learning Experiences. San Francisco: Pfeiffer. Driskell, J., B. Olmstead, and E. Salas. 1933. “Task Cues, Dominance Cues, and Influence in Task Groups.” Journal of Applied Psychology 78 (1): 51–60. Druskat, V., and S. Wolff. 2001. “Building the Emotional Intelligence of Groups.” Harvard Business Review, 79 (3), 80–90. Dubois, D., and W. Rothwell. 2000. The Competency Toolkit. 2 vols. Amherst, MA: Human Resource Development Press. Dubois, D., and W. Rothwell. 1996. Developing the High-Performance Workplace Organizational Assessment Package: Administrator’s Handbook and Data Collection Instrument. Amherst, MA: Human Resource Development Press. Dye, K. 1997. “Message Design: A Key to Effective Instructional Materials.” http:// kathydye.com/IDEAS/MessageDesign2.pdf. Eckel, N. 2010. “Collaborating with Subject Matter Experts.” T&D 64 (3): 76. Egan, K. 1978. “What Is Curriculum?” Curriculum and Inquiry 8 (1): 65–72. Ellington, H. 1985. Producing Teaching Materials: A Handbook for Teachers and Trainers. New York: Nichols. Ellis, P. F., and K. D. Kuznia. 2014. “Corporate eLearning Impact on Employees.” Global Journal of Business Research 8(4): 1–15. Emelo, R. 2012. “Building a Social Learning Strategy.” Training Journal (November). www.trainingjournal.com. Employment and Disability Institute at the Cornell University ILR School. 2012. 2012 Disability Status Report. www.disabilitystatistics.org. Erikson, E. 1959. Identity and the Life Cycle. New York: International Universities Press. Fiehl, S. 2012. “Integrating Social Media, Workplace Learning and eLearning for Development of Soft Skills.” Global Focus: The EFMD Business Magazine; 2012 Special Supplement 1 (6): 25–28. Finley, M. 2002. “All for One, but None for All? Why CEOs Make Lousy Team Players.” Across the Board 39 (1): 45–48. Fitz-Enz, J. 1984. How to Measure Human Resource Management. New York: McGraw-Hill. Fitz-Enz, J. 2000. The ROI of Human Capital. New York: AMACOM. Flanagan, J. 1954. “The Critical Incident Technique.” Psychological Bulletin 51: 327–358. Foshay, W., Silber, K., and Westgaard, O. 1986. Instructional Design Competencies: The Standards. Iowa City, IA: International Board of Standards for Training, Performance and Instruction.

References

371

Francis, D., and M. Woodcock. 1990. Unblocking Organizational Values. Glenview, IL: Scott-Foresman. Gagné, R. M., and L. J. Briggs. 1979. Principles of Instructional Design. New York: Holt, Rinehart and Winston. Gagné, R., L. Briggs, and W. Wager. 1992. Principles of Instructional Design. 4th ed. Fort Worth, TX: Harcourt Brace Jovanovich. Gagné, R., and K. Medsker. 1996. The Conditions of Learning: Training Applications. Fort Worth, TX: Harcourt Brace. Gagné, R., W. Wager, K. Golas, and J. Keller 2005. Principles of Instructional Design. 5th ed. Belmont, CA: Wadsworth. Galbraith, J. 2014. Designing Organizations: Strategy, Structure, and Process at the Business Unit and Enterprise Levels. 3rd ed. San Francisco: Jossey-Bass. Gartner Hype Cycle. www.gartner.com/technology/research/methodologies/hypecycle.jsp. Gendelman, J. 2009. “Aligning Business Needs and Instructional Assets (Recycling Instructional Assets).” Performance Improvement 48 (1): 25–36. Getting, Brian. 2007. “Basic Definitions: Web 1.0, Web. 2.0, Web 3.0.” April 18. www.practicalecommerce.com/articles/464-Basic-Definitions-Web-1-0-Web-2-0Web-3-0. Gillies, D. 1991. “Fine-Tuning Nurse Management Education through Formative Evaluation.” Journal of Continuing Education in the Health Professions 11 (3): 229–242. Green, Michael. 2011. “Better, Smarter, Faster: Web 3.0 and the Future of Learning.” T&D 65 (4). Gregory, G. H., and C. Chapman. 2012. Differentiated Instructional Strategies: One Size Doesn’t Fit All. Corwin Press. Griffiths, T., and D. Guile. 2003. “A Connective Model of Learning: The Implications for Work Process Knowledge.” European Educational Research Journal 2 (1): 56–73. Gross, R. (Ed.). 1982. The Independent Scholar’s Handbook. Reading, MA: Addison-Wesley. Gross, R. 1977. The Lifelong Learner . New York: Simon & Schuster. Guerra-Lopez, I. J. 2008. Performance Evaluation. San Francisco: John Wiley & Sons. Guillot, T. 2002. “Team Building in Virtual Environment.” Info-Line, No. 250205. Alexandria, VA: The American Society for Training and Development. Hannum, W. 2013. “Questioning Needs Assessment.” Educational Technology 53 (6): 29–34. Harless, J. 1985. “Performance Technology and Other Popular Myths.” Performance and Instruction Journal 24 (6): 4–6. Harless, J. 1986. “Guiding Performance with Job Aids.” In Introduction to Performance Technology, edited by M. Smith. Washington, DC: National Society for Performance and Instruction. Harris, P., and P. Castillo. 2002. Instructional Design for WBT. Info-Line, No. 250202. Alexandria, VA: The American Society for Training and Development. Hatcher, T. 2002. “Speaking of Ethics.” T&D 56: 2. Havighurst, R. 1970. Developmental Tasks and Education. 2nd ed. New York: McKay. Heideman, J. 1993. “The Team Approach to Formative Evaluation.” Technical and Skills Training 4 (3): 9–12.

372

References

Heinich, R., M. Molenda, J. Russell, and S. Smaldino. 2002. Instructional Media and Technologies for Learning . 7th ed. Englewood Cliffs, NJ: Prentice-Hall. Hensey, M. 1983. “Conflict: What It Is and What It Can Be.” In Conflict Resolution Technology, edited by D. Cole. Cleveland, OH: Organization Development Institute. Herbart, J. 1898. The Application of Psychology to the Science of Education. (Beatrice C. Mulliner, trans.) New York: Charles Scribner’s Sons. Holstein, J., and J. Gubrium (Eds.). 2001. Handbook of Interview Research: Context and Method. Thousand Oaks, CA: Sage. Houle, C. 1961. The Inquiring Mind. Madison, WI: University of Wisconsin Press. Huczynski, A. 1983. Encyclopedia of Management Development Models. London: Gower. Isaac, S., and W. Michael. 1984. Handbook of Research and Evaluation for Education and the Behavioral Sciences. 2nd ed. San Diego, CA: EDITS. Islam, K. 2006. Developing and Measuring Training the Six Sigma Way: A Business Approach to Training and Development. San Francisco: Pfeiffer. Jacobs, R. 1988. Effects of Feedback for Training and Development: Selected Research Abstracts. Columbus, OH: College of Education, Ohio State University. (ED 305 464) Johnson, N. 2010. Simply Complexity: A Clear Guide to Complexity Theory. Reprint edition. London: OneWorld Publications. Johnson, S. 1983. “Critical Incident.” In Human Resource Development: The Theory and Practice of Need Assessment, edited by F. Ulschak. Reston, VA: Reston Publishing. Jonassen, D., S. Grabinger, and N. Harris. 1990. “Analyzing and Selecting Instructional Strategies and Tactics.” Performance Improvement Quarterly 3 (2): 29–47. Kalaiselvan, K., and K. Naachimuthu. 2011. “A Synergistic Model to Training and Development.” Indian Journal of Industrial Relations 47 (2): 366–379. Kanfer R. 1990. “Motivation and Individual Differences in Learning: An Integration of Developmental, Differential and Cognitive Perspectives.” Learning and Individual Differences 2: 221–239. Kapp, K. 2012. The Gamification of Learning and Instruction: Game-based Methods and Strategies for Training and Education. San Francisco: Pfeiffer. Kasraie, N., and E. Kasraie. 2010. “Economies of Elearning in the 21st Century.” Contemporary Issues in Education Research 3 (10): 57–62. Katz, D., and R. Kahn. 1978. The Social Psychology of Organizations. 2nd ed. New York: John Wiley & Sons. Kaufman, R., and M. L. Bernardez. 2012. “Human Performance Technology and Its Future.” Performance Improvement Quarterly 25 (1): 5–11. Kaufman, R., and F. English. 1979. Needs Assessment: Concept and Application. Englewood Cliffs, NJ: Educational Technology Publications. Kaufman, R. 1986. “Assessing Needs.” In Introduction to Performance Technology, edited by M. Smith. Washington, DC: National Society for Performance and Instruction. Kaufman, R., J. Keller, and R. Watkins. 1995. “What Works and What Doesn’t: Evaluation Beyond Kirkpatrick.” Performance and Improvement 35 (2): 8–12. Kaufman, R., and I. Guerra-Lopez. 2013. Needs Assessment for Organizational Success. American Society for Training and Development. Kaufman, R., S. Thiagarajan, and P. MacGillis. The Guidebook for Performance Improvement: Working with Individuals and Organizations. San Francisco: Jossey-Bass. Keller, J. 2006. “What Is Motivational Design?” http://arcsmode.ipower.com/pdf/ Motivational%20Design%20Rev%20060620.pdf.

References

373

Keller, J. 2010. Motivational Design for Learning and Performance: The ARCS Model Approach. New York: Springer. Kemmerer, F., and S. Thiagarajan. 1989. “What Is an Incentive System?” Performance and Instruction 28 (3): 11–16. Kemp, J. 1985. The Instructional Design Process. New York: HarperCollins. Khan, S. 2012. The One World Schoolhouse: Education Reimagined. New York: Twelve Hachette Book Group. King, S., M. King, and W. Rothwell. 2000. The Complete Guide to Training Delivery: A Competency-Based Approach. New York: AMACOM. Kinlaw, D. 1993. Team-Managed Facilitation: Critical Skills for Developing Self-Sufficient Teams. San Francisco: Pfeiffer. Kirkpatrick, D. 1959. “Techniques for Evaluating Training Programs.” Journal of the American Society of Training Directors 13 (3–9): 21–26. Kirkpatrick, D. 1960. “Techniques for Evaluating Training Programs.” Journal of the American Society of Training Directors 14 (13–18): 28–32. Kirkpatrick, D. 1978. No-Nonsense Communication. 2nd ed. Elm Grove, WI: K & M Publishers. Kirkpatrick, D. 1985. How to Manage Change Efficiently: Approaches, Methods, and Case Examples. San Francisco: Jossey-Bass. Kirkpatrick, D. 1996. Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler. Kirkpatrick, D., and J. Kirkpatrick. Evaluating Training Programs: The Four Levels. 3rd ed. San Francisco: Berrett-Koehler. Kirkpatrick, J. and W. Kirkpatrick. 2015. “Do More with Less.” Chief Learning Officer (March 4). www.clomedia.com/articles/6118-do-more-with-less. Kiron, D., D. Palmer, A. N. Phillips, and R. Berkman. 2013. “Social Business Shifting Out of First Gear.” MIT Sloan Management Review Research Report, 1–28. Klein, J. D., and S. Jun. 2014. “Skills for Instructional Design Professionals.” Performance Improvement 53 (2): 41–46. Knowles, M. 1980. The Modern Practice of Adult Education: Andragogy Versus Pedagogy. New York: Association Press. Knowles, M. 1984. The Adult Learner: A Neglected Species. 3rd ed. Houston, TX: Gulf. Knowles, M., R. Swanson, and E. Holton. 2005. The Adult Learner: The Definitive Classic in Adult Education and Human Resource Development. 6th ed. San Francisco: Berrett-Koehler. Knowles, M. 1975. Self-Directed Learning: A Guide for Teachers and Learners. New York: Cambridge Book Company. Knowles, M. 1986. Using Learning Contracts: Practical Approaches to Individualizing and Structuring Learning . San Francisco: Jossey-Bass. Knox, A. 1977. Adult Development and Learning . San Francisco: Jossey-Bass. Ko, S., and S. Rossen. 2001. Teaching Online: A Practical Guide. New York: Houghton-Mifflin. Korkmaz, N., and E. Boling. 2014. “Development of Design Judgment in Instructional Design: Perspectives from Instructors, Students, and Instructional Designers.” Design in Educational Technology, 161–184.

374

References

Koszalka, T. A., D. F. Russ-Eft, and R. Reiser. 2013. Instructional Designer Competencies. The Standards. 4th ed. Charlotte, NC: Information Age. Krieger, G. 1994. “Constructing and Validating Tests.” In Human Resources Management and Development Handbook (3rd ed.), edited by W. Tracey. New York: AMACOM, 1239–1252. Krueger, R., and M. A. Case. 2000. Focus Groups: A Practical Guide for Applied Research. 3rd ed. Newbury Park, CA: Sage. Kuchinke, K., S. Aragon, and K. Bartlett. 2001. “Online Instructional Delivery.” Performance Improvement 40 (1): 19–27. Laird, D. 1985. Approaches to Training and Development. 2nd ed. Reading, MA: Addison-Wesley. Langdon, D., K. Whiteside, and M. McKenna. (Eds.). 1999. Intervention Resource Guide: 50 Performance Improvement Tools. San Francisco: Jossey-Bass. Lee, W., and R. Mamone. 1995a. “Design Criteria That Make Tests Objective.” Journal of Instruction Delivery Systems (Summer), 18–22. Lee, W., and R. Mamone. 1995b. Handbook of Computer Based Training: Assessment, Design, Development, Evaluation. Englewood Cliffs, NJ: Educational Technology. Lewis, T., and D. Bjorkquist. 1992. “Needs Assessment—A Critical Reappraisal.” Performance Improvement Quarterly 5 (4): 33–54. Levinson, D. 1978. The Seasons of a Man’s Life. New York: Knopf. Lewin, T. 2013. “Universities Abroad Join Partnerships on the Web.” New York Times (February 20). www.nytimes.com/2013/02/21/education/universities-abroadjoin-mooc-course-projects.html?_r=0. Lewis, T., and D. Bjorkquist. 1992. “Needs Assessment—A Critical Reappraisal.” Performance Improvement Quarterly 5 (4): 33–54. Likert, R. 1967. The Human Organization. New York: McGraw-Hill. Lipman, V. 2014. “71% of 16-to-24-Year-Olds Want ‘Wearable Tech.’ Why Don’t I Even Want to Wear a Watch?” (September 22). www.forbes.com/sites/victorlipman/ 2014/09/22/71-of-16-24s-want-wearable-tech-why-dont-i-even-want-to-wear-awatch/. Littlejohn, A. (Ed.). 2003. Reusing Online Resources: A Sustainable Approach to E-Learning . Hove, United Kingdom: Psychology Press. Mager, R. 1997a. Goal Analysis: How to Clarify Your Goals So You Can Actually Achieve Them. 3rd ed. Atlanta, GA: The Center for Effective Performance. Mager, R. F. 1988. Preparing Instructional Objectives. Belmont, CA: David S. Lake. Mager, R. 1997b. Preparing Instructional Objectives: A Critical Tool in the Development of Effective Instruction. 3rd ed. Atlanta, GA: The Center for Effective Performance. Mager, R., and P. Pipe. 1999. Analyzing Performance Problems or ‘You Really Oughta Wanna’ . 3rd ed. Atlanta, GA: The Center for Effective Performance. Malasky, E. 1984. “Instructional Strategies: Nonmedia.” In The Handbook of Human Resource Development, edited by L. Nadler, 9.1–9.31. New York: Wiley-Interscience. Mancini, J. 2014. “In a Cloud and Mobile World—Making Collaboration Less Confusing.” KM World 3 (August 29): 14. Marker, A., S. W. Villachica, D. Stepich, D. Allen, and L. Stanton. 2014. “An Updated Framework for Human Performance Improvement in the Workplace: The Spiral HPI Framework.” Performance Improvement 53 (1): 10–23.

References

375

Marrelli, A. F. 2005a. “Critical Incidents.” Performance Improvement 44 (November/December): 10. Marrelli, A. F. 2005b. “Process Mapping.” Performance Improvement 44 (May/June): 5. Marrelli, A. F. 2007. “Unobtrusive Measures.” Performance Improvement 46 (October): 9. Marrow, A. 1972. The Failure of Success. New York: AMACOM. Mattox, J. R. 2013. Net Promoter for Learning: Measuring and Improving Training Effectiveness? Inside Learning Technologies Magazine (October). http:// knowledgeadvisors.com/wp-content/uploads/2014/05/NPS-WP1-JM.pdf. Mayer, R. E. 2008. “Applying the Science of Learning: Evidence-Based Principles for the Design of Multimedia Instruction.” American Psychologist (November): 760–769. McKeown, J. O., and S. K. Howard. 2012. “The Effect of Delivery Method of Instructional Materials to Meet Learning Goals and Objectives in Online and Open Learning Environments.” The Next Generation of Distance Education, 85–96. McLagan, P. 2003. “The Change-Capable Organization.” Training and Development, 57(1), 50-58. McLinden, D., O. Cummings, and S. Bond. 1990. “A Comparison of Two Formats for an Instructor’s Guide.” Performance Improvement Quarterly 3 (1): 2–13. Meacham, M. 2015. Brain Matters: How to Help Anyone Learn Anything Using Neuroscience. Lexington, KY. Meister, Jeanne. 2013. “Gamification in Leadership Development: How Companies Use Gaming to Build Their Leadership Pipeline.” September 30. www.forbes.com/ sites/jeannemeister/2013/09/30/gamification-in-leadership-development-howcompanies-use-gaming-to-build-their-leader-pipeline/. Merrill, M. 2002. “A Pebble-in-the-Pond Model for Instructional Design.” Performance Improvement 41 (7): 41–46. Merrill, M. D. 2015. “A Pebble in a Pond Model for Instructional Design.” Performance Improvement 54 (1): 42–48. Merrill, M. D., L. Drake, M. J. Lacy, J. Pratt, and ID2 Research Group. 1996. Reclaiming Instructional Design.” Educational Technology 36 (5): 5–7. http:// mdavidmerrill.com/Papers/Reclaiming.PDF, i. Merrill, M., Z. Li, and M. Jones. 1990. “Limitations of First Generation Instructional Design.” Educational Technology (January), 7–11. Moore, J. 2008. “Web 2.0’s Defining Moment.” Better Investing 58 (2): 24–26. Moore, K. D. 2014. Effective Instructional Strategies: From Theory to Practice. Thousand Oaks, CA: Sage. Moore, S. 1998. “Time-to-Learning .” Digital Equipment Corporation. Moran, B. 2011. “Valuing eLearning.” Training and Development in Australia 38 (4): 34–36. Moseley, J., and J. Dessinger. 2007. Training Older Workers and Learners: Maximizing the Performance of an Aging Workforce. San Francisco: Pfeiffer. Muller, N., and V. Roberts. 2010. “Seven Cures to Skipping the Needs Assessment.” T&D (March): 64–63. Newstrom, J. 1980. “Evaluating the Effectiveness of Training Methods.” Personnel Administrator (January): 55–60. Newstrom, J., and J. Lilyquist. 1979. “Selecting Needs Analysis Methods.” Training and Development 33 (10): 52–56. Nichols, R. 1957. Successful Management. New York: Doubleday.

376

References

Odenwald, S. 1993. Global Training: How to Design a Program for the Multinational Corporation. Homewood, IL: Business One Irwin. Okes, D. 2008. “The Human Side of Root Cause Analysis.” The Journal for Quality and Participation (October 31): 3. Oluwafisayo. 2010. “Constructivism and Web 2.0 in the Emerging Learning Era: A Global Perspective.” Journal of Strategic Innovation and Sustainability 6 (4): 17–27. Osak, M. 2014. “Gamification Newest Tool in Corporate Training Arsenal.” Business Education. January 9. http://business.financialpost.com/executive/businesseducation/gamification-newest-tool-in-corporate-training-arsenal. Pepitone, J. S. 2000. Human Performance Consulting . Houston, TX: Gulf. Pershing, J. L. 2006. “A Commentary on Quantitative and Qualitative Methods.” In Handbook of Human Performance Technology. San Francisco: International Society for Performance Improvement. Phillips, J. 2011. Return on Investment in Training and Performance Improvement Programs. 2nd ed. London: Routledge. Phillips, J., and P. Phillips. 2008. Beyond Learning Objectives: Develop Measurable Objectives That Link to the Bottom Line. Alexandria, VA: ASTD Press. Phillips, J. J., and P. P. Phillips. 2010a. Measuring for Success. What CEOs Really Think About Learning Investments. Alexandria: ASTD. Phillips, J. J., and P. P. Phillips. 2010b. “How Executives View Learning Metrics by Patti and Jack Phillips.” CLO Magazine (December). http://clomedia.com/articles/ view/how-executives-view-learning-metrics/2. Phillips, P. (Ed.). 2010. ASTD Handbook of Measuring and Evaluating Training . Alexandria, VA: ATD Press. Prusak, L., and D. Cohen. 2001. “How to Invest in Social Capital.” Harvard Business Review 79 (6): 86–93. Pucel, D. 1989. Performance-Based Instructional Design. New York: McGraw-Hill. Putman, S. 2014. “Prototyping an E-Learning Interface.” T&D 68 (5): 26–27, 29. Rae, L. 1986. How to Measure Training Effectiveness. New York: Nichols. Reichheld, F. 2006. www.netpromoter.com/why-net-promoter/know. Reigeluth, C. 1999a. Instructional Design Theories and Models: A New Paradigm of Instructional Theory. Mahwah, NJ: Lawrence Erlbaum Associates. Reigeluth, C. (Ed.). 1999b. Instructional-Design Theories and Models: A New Paradigm of Instructional Theory. Vol. 2. London: Lawrence Erlbaum Associates. Reigeluth, C. (Ed.). 2008. Instructional-Design Theories and Models. Vol. 3. London: Lawrence Erlbaum. Reigeluth, C. M., and A. A. Carr-Chellman. 2009. Instructional-Design Theories and Models. New York: Routledge. Reiser, R. A. 2001. “A History of Instructional Design and Technology: Part II: A History of Instructional Design.” Educational Technology, Research and Development 49 (2): 57–67. Reiser, R. A., and J. V. Dempsey. 2011. Trends and Issues in Instructional Design and Technology. 3rd ed. Boston, MA: Pearson Education. Reynolds, A., and R. Anderson. 1992. Selecting and Developing Media for Instruction. New York: Van Nostrand Reinhold. Rich, B., J. LePine, and E. Crawford. 2010. “Job Engagement: Antecedents and Effects on Job Performance.” Academy of Management Journal 53: 617–635.

References

377

Richey, R., D. Fields, and M. Foxon. 2001. Instructional Design Competencies: The Standards. 3rd ed. Syracuse, NY: ERIC Clearinghouse on Information and Technology. Richey, R., J. Klein, and M. Tracey. 2010. The Instructional Design Knowledge Base: Theory, Research, and Practice. London: Routledge. Richman, R., and B. Kirlin. 2015. The Culture Blueprint: A Guide to Building the High-Performance Workplace. Culture Hackers. Rokeach, M. 1973. The Nature of Human Values. New York: The Free Press. Romiszowski, A. 1981. Designing Instructional Systems: Decision Making in Course Planning and Curriculum Design. New York: Nichols. Rooney, J. J., and L. N. Vanden Heuvel. 2004. “Root Cause Analysis for Beginners.” Quality Progress 37 (July ): 7. Ross School of Business, University of Michigan. 2013. “How to Write a Business Case Study.” November 23. http://globalens.com/DocFiles/PDF/cases/Preview/ GL1429140P.pdf. Rossett, A. 1987. Training Needs Assessment. Englewood Cliffs, NJ: Educational Technology Publications. Rothwell, W. 2002. The Workplace Learner: How to Align Training Initiatives with Individual Learning Competencies. New York: AMACOM. Rothwell, W. 2003. A Survey about Current Issues in Instructional Design. Unpublished report on survey results. University Park, PA: The Pennsylvania State University. Rothwell, W. 2008. Basics of Adult Learning . Alexandria, VA: ASTD Press. Rothwell, W. 2012. “13 Tips for Training in Other Countries.” T&D 66 (5): 38–42. Rothwell, W. 2015. Beyond Training and Development. 3rd ed. Amherst, MA: HRD Press. Rothwell, W. (Ed.). 2015. Organization Development Fundamentals: Managing Strategic Change. Alexandria, VA: ATD Press. Rothwell, W., R. Alzhahmi, C. Baumgardner, O. Buchko, W. Kim, J. Myers, and N. Sherwani. 2014. Creating Engaged Employees: It’s Worth the Investment. Alexandria, VA: ASTD Press. Rothwell, W., B. Benscoter, T. Park, B. Woocheol, and A. Zaballero. 2014. Performance Consulting: Applying Performance Improvement in Human Resource Development. Hoboken, NJ: John Wiley & Sons. Rothwell, W., M. Butler, C. Maldonado, D. Hunt, K. Peters, J. Li, and J. Stern. 2006. Handbook of Training Technology: An Introductory Guide to Facilitating Learning with Technology—From Planning through Evaluation. San Francisco: Pfeiffer & Co. Rothwell, W., and P. Chee. 2015. The Leader’s Daily Role in Talent Management. New York: McGraw-Hill. Rothwell, W., and P. Cookson. 1997. Beyond Instruction: Comprehensive Program Planning for Business and Education. San Francisco: Jossey-Bass. Rothwell, W. and J. Graber. 2010. Competency-Based Training Basics: A Complete How-To Guide. Alexandria, VA: ASTD Press. Rothwell, W., J. Graber, D. Dubois, A. Zaballero, C. Haynes, A. Alkhalaf, and S. Sager. 2015. The Competency Toolkit. 2nd ed. 2 vols. Amherst, MA: HRD Press. Rothwell, W., C. Hohne, and S. King. 2000. Human Performance Improvement. Houston, TX: Gulf Publishing.

378

References

Rothwell, W., R. Jackson, S. Knight, and J. Lindholm, with W. Wang and T. Payne. 2005. Career Planning and Succession Management: Developing Your Organization’s Talent—for Today and Tomorrow. Westport, CT: Greenwood Press. Rothwell, W., R. Jackson, C. Ressler, and M. Jones. 2015. Career Planning and Succession Management: Developing Your Organization’s Talent—for Today and Tomorrow. 2nd ed. Westport, CT: Greenwood Press. Rothwell, W., M. Jones, M. Kirby, and F. Loomis (Eds.). 2012. Talent Management: A Step-by-Action-Oriented Guide Based on Best Practice. Amherst, MA: Human Resource Development Press. Rothwell, W., and H. Kazanas. 1994a. Human Resource Development: A Strategic Approach (rev. ed.). Amherst, MA: Human Resource Development Press. Rothwell, W., and H. Kazanas. 1994b. Improving On-the-Job Training . San Francisco: Jossey-Bass,. Rothwell, W., and H. Kazanas. 2004. Improving On-the-Job Training . 2nd ed. San Francisco: Pfeiffer. Rothwell, W., and H. Kazanas. 2003. The Strategic Development of Talent. [Second edition of Human Resource Development: A Strategic Approach.] Amherst, MA: Human Resource Development Press. Rothwell, W., and J. Lindholm. 1999. “Competency Identification, Modeling, and Assessment in the USA.” International Journal of Training and Development 3 (2): 90–105. Rothwell, W., J. Lindholm, and W. Wallick. 2003. What CEOs Expect from Corporate Training: Building Workplace Learning and Performance Initiatives That Advance Organizational Goals. New York: AMACOM. Rothwell, W., and H. Sredl. 2000. The American Society for Training and Development Reference Guide to Workplace Learning and Performance. 3rd ed. 2 vols. Amherst, MA: Human Resource Development Press. Rothwell, W., J. Stavros, and R. Sullivan. (Eds.). In press. Practicing Organization Development: A Guide for Consultants. 4th ed. San Francisco: Pfeiffer. Rothwell, W., H. Sterns, D. Spokus, and J. Reaser. 2008. Working Longer: New Strategies for Managing, Training, and Retaining Older Employees. New York: AMACOM. Rothwell, W., A. Zaballero, T. Asino, J. Briskin, A. Swaggerty, and S. Bienert. 2015. Skills, Trends, and Challenges in Instructional Design. Alexandria, VA: ATD Press. Rummler, G. 1986. “Organizational Redesign.” In Introduction to Performance Technology, edited by M. Smith. Washington, DC: The National Society for Performance and Instruction. Rummler, G. 1983. “Human Performance Problems and Their Solutions.” In The Training and Development Sourcebook, edited by L. Baird, C. Schneier, and D. Laird, 7–18. Amherst, MA: Human Resource Development Press. Rummler, G. 1976. “The Performance Audit.” In Training and Development Handbook: A Guide to Human Resource Development (2nd ed.), edited by R. Craig. New York: McGraw-Hill, 14–3. Rush, D. K., and S. J. Schmitz. 2009. “Universal Instructional Design. Engaging the Whole Class.” May. http://works.bepress.com/suzanne_schmitz/1/. Russell, J., and B. Blake. 1988. “Formative and Summative Evaluation of Instructional Products and Learners.” Educational Technology 28 (9): 22–28.

References

379

Schein, E. 1969. Process Consultation: Its Role in Organization Development. Reading, MA: Addison-Wesley. Schein, E. 1985. Organizational Culture and Leadership: A Dynamic View. San Francisco: Jossey-Bass. Schwahn, C., and B. McGarvey. 2012. Inevitable: Mass Customized Learning: Learning in the Age of Empowerment. CreateSpace Independent Publishing Platform. Schwarz, R. 1994. The Skilled Facilitator: Practical Wisdom for Developing Effective Groups. San Francisco: Jossey-Bass. Seels, B. B., and R. C. Richey. 1994. Instructional Technology: The Definition and Domains of the Field. Bloomington, IN: Association for Educational Communication and Technology. Senge, P. 1990. The Fifth Discipline: The Art and Practice of the Learning Organization. New York: Currency Doubleday. Sheehy, G. 1974. Passages: Predictable Crises of Adult Life. New York: Dutton. Siemens, G. “Connectivism: A Learning Theory for the Digital Age,” International Journal of Instructional Technology and Distance Learning 2 (1): 1–6. Silber, K., and W. Foshay 2009. Handbook of Improving Performance in the Workplace, Instructional Design and Training Delivery. Hoboken, NJ: John Wiley & Sons. Sinha, S. 2012. “Motivating Students and the Gamification of Learning.” Huffington Post, February 14. www.huffingtonpost.com/shantanu-sinha/motivating-studentsand-t_b_1275441.html. Sousa, D. A. 2011. How the Brain Learns. 6th ed. Thousand Oaks, CA : David A. Sousa. Steele, F. 1973. Physical Settings and Organization Development. Reading, MA: Addison-Wesley. Stolovitch, H. D. 2015. “Human Performance Technology: Research and Theory to Practice.” Performance Improvement 54 (3): 37–46. Stolovitch, H. D., and E. J. Keeps. 1992. Handbook of Human Performance Technology. San Francisco: Jossey-Bass. Sugar, W., B. Hoard, A. Brown, and L. Daniels. 2011. “Identifying Multimedia Production Competencies and Skills of Instructional Design and Technology Professionals: An Analysis of Recent Job Postings.” Journal of Educational Technology Systems 40 (3): 227–249. Swanson, R. A. 1994. Analysis for Improving Performance. San Francisco: Berrett-Koehler Publishers. Taflinger, R. 2011. “Introduction to Research.” http://public.wsu.edu/~taflinge/ research.html. Taylor, K. 2006. “Brain Function and Adult Learning: Implication for Practice.” In The Neuroscience of Adult Learning , edited by S. Johnson & Kathleen Taylor. San Francisco, CA. Jossey-Bass, No. 110. Tenopyr, M. 1996. “Testing.” In The ASTD Training and Development Handbook: A Guide to Human Resource Development (4th ed.), edited by R. Craig. New York: McGraw-Hill, 342–356. Terrell, S. R. 2012. “Mixed-Methods Research Methodologies.” The Qualitative Report 17 (1): 254–280. www.nova.edu/ssss/QR/QR17-1/terrell.pdf. Tessmer, M. 1990. “A Neglected Stage of Instructional Design.” Educational Technology Research and Development 38 (1): 55–64.

380

References

Tessmer, M. 1994. “Formative Evaluation Alternatives.” Performance Improvement Quarterly 7 (1): 3–18. Tessmer, M., and R. Richey, R. 1997. “The Role of Context in Learning and Instructional Design.” Educational Technology Research and Development 45 (2): 85–115. Thalheimer, W. 2013. “Net Promotor Score—Maybe Fine for Marketing, Stupid for Training.” www.willatworklearning.com/2013/12/net-promoter-score-maybe-finefor-marketing-stupid-for-training.html. Thiagarajan, S. 1991. “Formative Evaluation in Performance Technology.” Performance Improvement Quarterly 4 (2): 22–34. Thompson, S. 2002. Sampling . 2nd ed. New York: John Wiley & Sons. Thompson, T., D. Felce, and F. Symons (Eds.). 1999. Behavioral Observation: Technology and Applications in Developmental Disabilities. Baltimore, MD: Paul H. Brookes. Thorndike, E., and R. Woodworth. 1901a. “The Estimation of Magnitudes.” Psychological Review 8: 384–395. Thorndike, E., and R. Woodworth. 1901b. “Functions Involving Attention, Observation, and Discrimination.” Psychological Review 8: 553–564. Thorndike, E., and R. Woodworth. 1901c. “The Influence of Improvement in One Mental Function upon the Efficiency of Other Functions.” Psychological Review 8: 247–261. Thornton, G. 1992. Assessment Centers in Human Resource Management. Reading, MA: Addison-Wesley. Torres, R., et al. 2005. Evaluation Strategies for Communicating and Reporting . Thousand Oaks, CA: Sage. Tosti, D., and S. Jackson. 1992. “Influencing Others to Act.” In Handbook of Human Performance Technology: A Comprehensive Guide for Analyzing and Solving Performance Problems in Organizations, edited by H. Stolovitch and E. Keeps, 551–563. San Francisco: Jossey-Bass. Towers Watson. 2013–2014. Change and Communication ROI—The 10th Anniversary Report. How the Fundamentals Evolve and the Best Adapt. Tough, A. 1979. The Adult’s Learning Projects. 2nd ed. Toronto, Ontario: Institute for Studies in Education. Training Industry. 2014. “Gamification.” www.trainingindustry.com/top-companieslisting/gamification/2014/2014-top-20-gamification-companies-list.aspx. Trees, L. 2014. “Content Management: Drivers of Effectiveness. Part 2: Tools and Technologies to Get the Job Done.” KM World. 23 (8): Pages 1–2. Vance, D. L. 2010. The Business of Learning . Windsor, CO. Poudre River Press. Van Grove, J. 2011. “Gamification: How Competition Is Reinventing Business, Marketing & Everyday Life.” Mashable (July 28). http://mashable.com/2011/07/ 28/gamification/. Vercoustre, A. M., and A. McLean, A. 2005. “Reusing Educational Material for Teaching and Learning: Current Approaches and Directions.” International Journal on E-learning 4(1): 57–68. Vroom, V. 1964. Work and Motivation. New York: John Wiley & Sons. Walton, R. 1968. Interpersonal Peacemaking: Confrontations and Third Party Consultation. Reading, MA: Addison-Wesley,

References

381

Watkins, R., and R. Kaufman. 1996. “An Update on Relating Needs Assessment and Needs Analysis.” Performance Improvement 35 (10): 10–13. Watkins, K., and V. Marsick. 1993. Sculpting the Learning Organization: Lessons in the Art and Science of Systemic Change. San Francisco: Jossey-Bass. Watson, D., and L. Llorens. 1997. Task Analysis: An Occupational Performance Approach. Bethesda, MD: American Occupational Therapy Association. Wedman, J. 2008. Needs Assessment Basics. http://needsassessment.missouri.edu/. Weisbord, M. 1993. “Diagnosing Your Organization: Six Places to Look for Trouble with or Without a Theory.” In Handbook of Organizational Consultation, edited by R. Golembiewski, 753–766. New York: Marcel Dekker. Whitney, D., D. Cooperrider, B. Kaplin, and A. Trosten-Bloom. 2001. Encyclopedia of Positive Questions, Volume I: Using AI to Bring Out the Best in Your Organization (Tools in Appreciative Inquiry Series, Volume 2). Lakewood, IL: Lakeshore Communications. Winiecki, D. J. 2009. Shadowboxing with Data: A Framework for Informing the Critical Analysis of Performance and Performance Measures. Performance Improvement 48 (2): 31–37. Wlodkowski, R. 1999. Enhancing Adult Motivation to Learn: A Comprehensive Guide to Teaching All Adults. Rev. ed. San Francisco: Jossey-Bass. Wolfe, P. 2010. Brain Matters: Translating Research into Classroom Practice. 2nd ed. Alexandria, VA: ASCD. Yang, D. 2013. “Are We MOOC’d Out?” Huffington Post (March 14). www .huffingtonpost.com/dennis-yang/post_4496_b_2877799.html. Zinger, D. 2014. “Game On: A Primer on Gamification for Managers” (May 8). www.td .org/Publications/Magazines/TD/TD-Archive/2014/05/Game-on-a-Primer-onGamification-for-Managers.

INDEX

A ABI Research, 346–347 Accuracy, 266 Adjustments, 287 Adoption, 285 Advanced competency, 215, 334 Advisory committee, 227 Affiliation, 320 After-action reviews, 333 Agenda, 248, 330 Align, 275 Allocating resources for project plan support establishing and using methods to reallocate funds, 339 managing multiple priorities and identifying/resolving project issues, 340

planning and monitoring equipment and facility, 339 Allocation of resources, 335 Ambiguous situations, 324–325 Apology, 330–331 Areas of improvement, 310 Asian cultures, 322 Assessment alignment goals and outcomes, 229–230 Association for Talent Development (ATD), 309 Association for Talent Development’s Membership Information Service, 246 Assumption clarification, 328 Attitude survey, 227 383

384

Index

Audit, 297 Authoring tool, 344

B Baker, H., 328 Balance, 270 Bandura, A., 348 Behavioral interviewing, 307 Behaviors examples test, 224 Behind the scenes, 278 Benefits of the change, 288 Best way, 297 Blended learning, 278 Brain ability, 358 Brain functioning, 360 Brain research, 360 Brainstorming, 318 Brainwave and cognitive functioning, 347 Bribing, 329 Brinkerhoff, R. O., 258–259 Budget, 337 Budget constraints, 282 Building H Performance Teams, 257 Bureaucratic organizations, 314 Business case, 299, 300 Business metrics, 257 Business skill for instructional design functions, 291–292 about, 291–292 contributing to organizational purpose and strategy, 295–299 instructional design for larger organization, 292–295

managing instructional design talent, 303–304 marketing communicating about instructional design, 313–314 navigating organizational cultural, structure, politics and processes, 314–316 standards implementing, 299–301

C Call center, 257 Career derailment, 318 Career planning, 312 Centers of Excellence (COE), 293 Centers of Expertise (COE), 293 Chain of command, 321 “Change-capable” organization, 320 Checklist, 297 Clarity, 266 Closed questions, 326 Closed-end items, 223 Coaching tools, 285 Cobbler’s children syndrome, 309 Coercing, 329 Collateral, 314 Commitment, 286, 331 Common way, 297 Communicating, 332 Communication, 287 Communication tool, 313 Community of Practice (CoP), 294 Competency, 303 Competency work, 292 Compliance, 286

Index

Comprehensive business, 300 Computer Age, 350 Conditions of rapport, 320 Conflict frequency of, 340 and friction, 327 management, 328–329 resolution approach, 329 symptoms of, 329 Conflicting demand, 339 Connectivism, 348 Content, 266, 327 Continuous improvement changes, 141 Contributing to organizational purpose and strategy, establishing standards of excellence and governance, 295–299 Control chart, 339 Control mechanism, 330 Controls, 297 Cooperrider, B., 327 Copyright issues, 352 Core ISD process, 270 Core mission, 295 Corporate COE, 294 Corporate Executive Board, 306, 308 Corporate planners, 332 Cost center, 313 Covey, Stephen R., 272 Creating a vision for change aligning learning and performance and organizational goals, 273 benefits of alignment and challengers, 274–275

385

engaging stakeholders and negotiation to achieve alignment, 275–276 Creativity and innovation, 257 Credibility, 228–229 Criterion-referenced tests (CRT), 238 Cross-cultural implications, 322 Cultural characteristics, 315 Cultural norm, 315 Culture current, 315 and key belief, 321 of the organization, 315 Customer service, 336 Customers and/or clients, 336

D Daily “huddle” meetings, 283 Damasio, A., 359 Data-based revision about, 263 revision types, 264 revisions to delivery or products/programs, 269–270 stakeholder revisions support, 267–268 Data collection approaches, data collection methods, 235–236 Data collection, initial, 337 Data collection methods and analysis, 236 criterion-reinforced tests (CRT), 238 extant data review, 239 focus groups, 236–237

386

interviews, 236 norm-reference tests NRT, 238 surveys or questionnaires, 237 tests, 237–238 Davidson, C., 348 Decentralized inconsistent approach, 296 Decision makers, 284, 299 experience with formative evaluation, 242 Delivery process, 264–265 Denying, 329 Depth of adoption, 286 Descartes’ Error (Damasio, M.), 359 Design project team, 334 Design skills, 311 Design standards, 297 Designing learning assessments designing learning introduction, 215–221 reliable and valid models, 228–229 Designing learning introduction background research, 222 deciding measuring,’ 220–221 deciding the purpose, 217 descriptive instrument, 222 drafting or modifying, 223–225 information sources, 218–220 instrument revising, 227 instrument with tracking means, 226 large group try out, 226 other performance measurement methods, 227–228

Index

performance measurements importance, 215 sequencing items, 226 small-group try out, 226 Developing Your Career workshop, 260 Development plan, 310, 312 Dialogue, 320 Different ways, 315 Diffusion, 272, 285, 287 Digital Age, 350 Diligent, 321 Direct feedback, 264 Direct observation, 256 Disabled workers, 244 Dissemination method, 262 planning for, 272

E Educational Testing Service, 222 Effective executive coaches, 320 Efficiencies, 296 Empathy practicing, 332 Employee engagement, 309 Encouraging participation, 332 Engagement, 268, 308 Engagement driver, 309 Engagement tactics, 319 Establishing rapport, 319 Ethical Awareness Decision Making, 284

Index

387

Evaluating instruction rights, 242 Evaluation design, 245 information purpose or purposes, 260 plan, 234 report elements, 261 report recipient, 260, 265 results, 260 Exemplars, 243 Exit interview, 227 Expert designers, 292 Expert reviews, 247 Expert status, 321 Experts, 246 Extant data review, 239 External assessment center, 227 External benchmarking, 304 External consultants, 321 External providers, 311 External resources, 301 Externally focused questions, 326

F Face-to-face interventions, 277 Face-to-face relationships, 320 Facilitator role, 280 Facilitators, 301 Fail miserably, 317 False assumptions, 320 Feedback, 287 Final word, 362–363 Final work product, 245

Flexibility, 314 Focus groups, 236–237 Follow-up actions, 299 Formal report, 251 Formative evaluation, 230, 234, 243–245 assumptions about, 239 decisions maker’s experience with, 242 terms defined about, 240 Formative evaluation plan, 234 Formative evaluation plan development, 233 determining purpose, objectives audience and subject, 240–241 information needs assessment, 241–242 management plan, 245–246 population studied and subject selected, 243–244 proper protocol consideration, 242–243 study design formulation, 244–245 variables importance identified, 244 Formative process evaluation, 240 Formative product evaluation, 240 Foshay, W., 320 “Four Levels,” 252 Fresh perspective, 321 Friction, 328

388

Index

Future experience, 225–226 preparation, 141–143

Human Resource Business Partners (HR BP), 293 Humor, 333 Hype cycle, 344

G Gamification (Van Grove, J.), 347 Gamification companies, 348 Gartner, H. C., 344 Geographic dispersion, 282 Global sourcing of content and design, open-sourced design and development, 352 Globalization, 343, 351–352 Governance, 295–297 structure, 336 Group discussion, 227 Group wandering, 330

H Hensey, M., 328 Hidden agendas, 332 High-potential workers, 243 Hiring, 304 Hiring decision, 307 Horizontal alignment, 274 How much focus, 240 HR model, 293 HRLC’s Employee Engagement Survey, 308 Human performance problem, 322

I ID practices, 353 ID Process, 233, 351, 352 ID support, low-cost sources, 352 Identifying and engaging stakeholders about, 318–319 how to establish rapport, 319–320 importance of rapport, 319–320 Immediate supervisors, 241–242 Impact of neuroscience research on instructional systems design about, 358–359 implications for instructional designers, 360–361 major advances in neuroscience research, 358–359 Implementation, 272, 276, 278, 287 Implementing an intervention, 278 Implementing instructional and noninstructional interventions about, 271–272 compliance vs. gaining commitment, 286–287 creating a vision for change, 272–276 encouraging and achieving adoption and buy-in, 285–286

Index

monitoring implementation, implementation, and diffusion to identify potential adjustments, 287 planning for diffusion of the intervention, 285 planning for implementation intervention of implementation and dissemination, 276–285 taking action on deviations or problems surfaced through monitoring, 288 Incompetence, 257 Industrial Age, 350 Ineffective conflict resolution, 329 Inevitable: Mass Customized Learning (Schwahn, C. and McGarvey, B.), 350 Informal learning, 355–356 Information gathered, 325 given, 320 withheld, 320 Information Age, 350 In-house politics, 321 Initial project goals, 340 Innovation ways, 315 Insiders, 320–321 The Instruction Designer Community of Practice, 294 Instruction designers, 317, 321 Instruction post-implementation, 234 Instructional design, 352

389

Instructional design experts (IDEs), 246 Instructional design for larger organization about, 292–293 community of practice, 293–295 embedded model, 293 human resources of expertise model, 293 Instructional design function, 292, 317 Instructional Design Level, 312 Instructional design practitioner, 311 Instructional design projects allocating resources for project plan support, 338–340 planning and management, 334–340 unique challenges posed by project management, 335–336 Instructional design role, 358 Instructional design standards, 296 Instructional design team, 317 Instructional design work, 318 Instructional designers, 221, 229, 241, 276, 292, 294, 321, 325, 330–332, 353 Instructional experience, 221 Instructional intervention, 301 Instructional package, 267 Integrated software platforms, 253

390

Intellectual and copyright issues, 363 Interaction(s), 322 with others, 317 returns, 329 Internal or external auditor, 297 Internal or external consultant, 320 Internal resources, 301 Internal salary analysis, 304 Internally focused questions, 326 Internet, 356 Interpersonal trust, 319 Inter-projectional dynamics, 333 Interventions, 276 Interventions instructional and noninstructional about, 233–234 expert reviews, 246–248 feedback from formative evaluation, 251 formative evaluation, 239–240 formative evaluation approaches, 246–251, 250–251 formative evaluation plan development, 240–246 group pretests and pilot tests, 250 management or executive rehearsals, 248 pretests and pilot tests individualized, 249–250 purpose definition of evaluation, 234–239

Index

report creating, 261–262 report dissemination, 262 reporting evaluation results, 259–261 summative evaluations, 251–259 Interview duration, 306 Interview Skill workshop, 278 Interviews, 236 Issue in proposal writing, 338 Item format, 224 ITunes, 350

J Job aids, 285 Just enough learning, 351 Just-in-time, 351

K Kaplin, B., 327 Kaufman, R., 24, 26 Key assumptions, 325 Key decision makers, 241–242, 331 Key questions, 218 Key stakeholders, 261 Khan Academy, 348 Kirkpatrick, D., 282, 332 Kirkpatrick alternatives, 282 Kirkpatrick model, 258 Kirkpatrick’s four levels level 1, 252–253 level 2, 254–255 level 3, 255–257 level 4, 257–258

Index

391

Kirkpatrick’s framework, 255 Knowledge, 254 Knowledge Advisors, 255 Knowledge tests, 237, 254, 321

L Labels, 242 Lateral career, 312 Lateral career progression, 312 Lateral moves, 312 Leadership, 308 Leadership competencies, 272 Leading Change program, 278 Learner progress, 282 Learner worker, 218 Learners, 229 Learning assets, 313 Learning experience, 353 Learning management system (LMS), 284 Learning objectives, 336 Learning theory, 348 Leveraging Inclusive Diversity program, 266 Lifelong learning, 356 Lipmann, V., 347 List of action steps, 337 Listening, working at, 327 Listening efficiency, 327 LMS, 285 LMS team, 269 Logistical plan, 272 Long-term authority, 335

M Macro trends, 343 Management plan, 245 Managing cross-functional teams about, 327–328 friction arise sources, 328 friction management, 328–329 people guided back on track, 330–331 people kept on track, 329–330 Managing instructional design talent career development, 312 developing instructional design talent, 309 engaging and returning design talent, 308–309 individual development planning, 310 job description for design consultant, 305–306 professional skills development, 310–311 recruiting instructional design talent, 304–307 Managing partnerships and collaborative relationships asking questions, 325–327 building relations between instruction designers and others, 322–327 identifying and engaging stakeholders, 318–319 instructional designer’s position power, 325

392

listening activity, 327 managing cross-functional teams, 327–331 project reviews, 333 securing commitment, 331–333 with stakeholders, 317 Marketing pieces, 313 Masie Center, 311 Massive Open Online Courses (MOOCS), 349, 354 Master scheduling, 339 Master trainer, 280 Mastering, 234 McGarvey, B., 350 McLagan, P., 320 Measure decision, 220 Measurement capability, 285 Measurements method, 221 Measurements of all kind, 215 Media popularity, 262 Meetings control regaining, 331 daily “huddle,” 328 topic of, 330 Methods, 236 Metrics, 215 Metrics That Matter platform, 255 Micro-level monitoring, 283 Microsoft Project, 338 Millennials, 356 Misunderstanding reduction, 325 Mobile devices, 346 Mobile learning, 362 Monitoring, 283, 287

Index

MOOCS. See Massive Open Online Courses (MOOCS) Moralizing, 329 Morgan, P., 328 Multiple data sources, 236 Multitask, 359 Mutual trust, 319

N Navigating Your Career workshop, 281 Negative emotions, 333 Negative side effects, 325–326 Negotiation, 276 Net Promoter Score (NPS), 253 Neuroscience, 358–359 New hire orientation, 345 New knowledge creation, 353 Noninstructional intervention, 301 Nontechnical interpersonal skills, 318 Norm-reference tests (NRT), 238 Norms of behavior, 316 Numeric data sources, 235 Numeric quantitative data, 235

O Observation, 237 Odenwald, S., 322 Off-the-job environment, 250 Onboarding, 307–308, 345 Online learning, 313 On-the-job performance change, 217 Open sources, 352

Index

393

Open-ended items, 223 Opportunities, 310 Organization culture, 243 Organization impact, 217 Organizational change, 288 Organizational metrics, 257 Organizational politics, 316 Organizational traditions, 321 Organizations line vs. staff, 335 Outsiders, 321 Ownership, sense of and commitment, 333

P Participants learning, 217 minds of, 330 reaction, 217, 265, 333 Participation encouraging, 332 Partnerships with external stakeholders, 317 Pay practices, 330 Pay-as-you-go model, 351 People on track, 330 Performance goals, 273, 336 measure, 220 measurements, 215 objectives, 218 problems, 325 standards, 334 statements, 292 Phillips ROI model, 258 Physiological responses, 347 Pilot tests, 230, 249–250

Planning for implementation intervention of implementation and dissemination aspects of implementation, 278–280 blended learning, 277 implementing instructional and noninstructional interventions, 276–285 learning management systems to monitor learner progress and course completion, 284–285 monitoring quality of implementation, dissemination and learn progress, 282–284 in-person interventions, 277 self-paced interventions, 277 train-the-trainer approach to dissemination, 280–282 Political dynamics, 316 Poor hiring decisions, 304 Poor planning, 336 Position power, 325 Post Millennials, 357 Post-implementation, 234 Post-tests, 238 Potential candidates, 304 Potential changes, 265 Predicting and future of learning and instructional design impact of neuroscience research on instructional systems design, 358–361 technological advances to improve delivery and design, 343 Pretest not train purpose, 249 Pretests, 237, 249

394

Prevailing culture, 315 Primary categories, 264 Priorities questions, 339 Problem performers, 243 Problems human performance, 322 performance, 325 performers, 243 root-cause, 337 significance of, 325–326 statement of, 337 surfaced through monitoring, 288 taking action on, 288 Product or program revisions, 264 Professional facilitator, 280 Professionals, 291 Project(s) deadlines, 339 goals, 336 managers, 335 out of control, 335 planning, 334 reviews, 333 Prospect client, 322 Prototyping, 270 Purpose definition of evaluation data collection approaches, 235–236 data collection methods, 236–239 evaluation data sources, 235–236 stakeholders in evaluation, 234–235

Index

Q Qualitative data, 236 Quality control department, 297 Quality of adoption, 286 Quantitative data, 235 Quantitative data sources, 235 Quest to Learn (Davison, C.), 348 Questions asking, 325–327 book full of, 327 closed, 326 externally focused, 326 of individuals or groups, 325 internally focused, 326 key, 218 menu, 220 priorities, 339 stakeholders, 268 tones of, 326 Quick answers, 353

R Ramp-up time, 282 Rapport, 320 Rapport establishment, 321 Real-life situations, 345 Recruiting, 304 Rehearsal, 248 Relevance deals, 266 Reliability categories, 228–229 Representatives from stakeholder groups, 336 of targeted learners, 242

Index

395

Resources, 301 Retention purposes, 347 Return-on-investment (ROI), 194, 258, 338 Revision types assessing and predicting efficacy, 265–266 input sources for potential revisions, 264 processes to improve outcomes, 265 reviewing need revisions, 265 revising to improve outcomes, 266–267 Revisions to delivery or products/programs, 264 data based revision, 269–270 evaluating revisions effectiveness, 270 revisions making in environment, 270 Risk mitigation, 288 Role of ID, 363 Root-cause problem, 282 Runaway costs, 334

S Scenarios, 299 Schedule and deliver, 282 Schein, Edgar, 331 Schwahn, C., 350 Securing commitment commitment importance, 331 commitment obtained, 332–333

Selection activities, 304 Self-directed learning, 356 Seminar providers, 311 Service provider, 313 Seven Habits of Highly Effective People (Covey, Stephen R.), 272 Sign-up system, 339 Silber, K., 320 Silos, 274. See also Deadly dozen silos expertise, 199–200 generational, 198 hidden, 203 intelligence, 201 national, 201–202 objective, 195 operational, 199 organizational, 197–198 responsibilities and accountability, 200 solution, 197 touchpoints, 203 Simulation, applications of, 346 Simulation technologies, 345 Skill acquisition, 254 Skill tests, 237 Skill-orientation test, 255 SMART goals, 336 SME, 319 Social ties, 321 Soft skills, 310, 317, 318 Software tools, 338 Special agenda, 248 Special needs, 222 Specialized information, 244

396

Specialized methods, 244 Speed of adoption, 286 Staffing list, 337 Stakeholders, 261, 267, 284 groups representatives from, 336 involvement, 286 key participation, 243 managing partnerships and collaborative relationships with, 317–319 preference, 218 questions, 268 Standard, 296 Standard simulation, 345 Standardization, 296 The Standards, 228, 234, 263, 271–272, 272, 291, 297, 299, 303, 317, 318 Starting point, 249 Statement of the problems, 337 Statement of the solutions, 337 Strategists, 229 Structure and processes, 172–174 Subject-matter experts (SMEs), 246, 301 Submitting, 329 Success Case Method (Brinkerhoff, R. O.), 258–259 Summative evaluations, 234, 239 about, 251–252 Kirkpatrick alternatives, 258 Kirkpatrick’s four levels, 252–258 Phillips ROI Model, 258 success case method, 258–259 Support, 275 allocating resources for project plan, 339–340

Index

and cooperation, 325 ID support, 352 stakeholder revisions, 267–268 Survey software applications, 253 Surveys or questionnaires, 237 Sustainability and adoption, 344

T Talent management, 303, 310 Tap-on-the-shoulder, 281 Target of alignment, 275 Targeted learners representatives, 242 Tasks essential, 310 Taylor, K., 362 Team members, 335 Technical skills, 310 Technological advances, 343 Technological advances to improve delivery and design future learner, 356–357 gamification, 347–348 global sourcing of content and design, 351–352 informal learning, 355–356 intellectual property and copyright issues, 352–353 mass customization of learning, 350–351 networked learning through virtual collaboration, 348–350 outsourcing instructional design, 351 proliferation of video-based learning, 354–355

Index

397

rapid deployment from need to solution, 353–354 simulation and virtual reality, 345–346 ubiquitous competing mobile, and wearable technology, 346–347 virtual delivery and scale, 344–345 Terms defined about, formative evaluation, 240 Test, 85 Third-party audits, 297 Timeline, 337 Timing, 243 Training needs, 330 Train-the-trainer method, 280, 288 Transfer of learning, 255 Trends, 343, 353 Trosten-Bloom, A., 327 Trust and rapport, 318 Trusting relationship, 319 Trustworthy, 321 Tryouts, 249

U Ubiquitous computing, 346 Under-performance, 318 Unique challenges posed by project management about, 335–336 establishing project scope and goals, 336–337 planning and management tools for instructional design plans, 338

writing proposals for instructional design projects, 337 writing proposals tricks of the trade, 337–338 University of British Columbia, 294

V Validity, 228 Van Grove, J., 347 Vendors, 321 Veterans, 243 Video-based learning, 354 Viral videos, 354 Virtual collaboration, 349 Virtual practitioners, 293 Virtual realities, 346, 362 Virtual reality games, 345 Virtual technologies, 277 Vocal participants, 330

W Wearable technologies, 362 Western culture, 322 Westgard, O., 320 Whitney, D., 327 Wide net, 287 Work assignments, 335 Working at listening, 327 Workshop providers, 311 Writing proposals, 335, 337

Y Yang, Dennis, 349 YouTube, 354