Technology and assessment strategies for improving student learning in chemistry 9780841231801, 084123180X, 9780841231818

Table of contents :
Content: Development of Scaffolded Online Modules To Support Self-Regulated Learning in Chemistry Concepts / Lawrie, Gwendolyn A., School of Chemistry & Molecular Biosciences, The University of Queensland, St Lucia, QLD 4072, Australia
Schultz, Madeleine, School of Chemistry, Physics and Mechanical Engineering, Queensland University of Technology, Brisbane, QLD 4001, Australia
Bailey, Chantal H., School of Chemistry & Molecular Biosciences, The University of Queensland, St Lucia, QLD 4072, Australia
Al Mamun, Md. Abdullah, School of Education, The University of Queensland, St Lucia, QLD 4072, Australia
Micallef, Aaron S., School of Chemistry, Physics and Mechanical Engineering, Queensland University of Technology, Brisbane, QLD 4001, Australia
Williams, Mark, School of Science & Health, Western Sydney University, Penrith, 2751 NSW, Australia
Wright, Anthony H., School of Education, The University of Queensland, St Lucia, QLD 4072, Australia / Improving Academic Reading Habits in Chemistry through Flipping with an Open Education Digital Textbook / McCollum, Brett M. / Using Web 2.0 Technology in Assessment of Learning in Chemistry: Drawing Threads between Teaching as Practice and Teaching as Research / Lawrie, Gwendolyn A. / Combining Educational Technologies for Student Engagement in the Chemistry Classroom / Redd, Ginger P., Department of Chemistry, North Carolina Agricultural and Technical State University, 1601 East Market Street, Greensboro, North Carolina 27411, United States
Redd, Thomas C., Department of Mathematics, North Carolina Agricultural and Technical State University, 1601 East Market Street, Greensboro, North Carolina 27411, United States
Lewis, Tracie O., Instructional Technology Services & Distance Education, North Carolina Agricultural and Technical State University, 1601 East Market Street, Greensboro, North Carolina 27411, United States
Gravely, Etta C., Department of Chemistry, North Carolina Agricultural and Technical State University, 1601 East Market Street, Greensboro, North Carolina 27411, United States / Evaluating the Use of LearnSmart and Connect in Introductory General Chemistry Classes: The Pros and Cons of an Online Teaching and Learning System / Venkateswaran, Rashmi / Faculty Goals, Inquiry, and Meaningful Learning in the Undergraduate Chemistry Laboratory / Bretz, Stacey Lowery
Galloway, Kelli Rush
Orzel, Joanna
Gross, Elizabeth / Exploring the Instructional Use of Contrasting Molecular Animations of a Redox Reaction / Kelly, Resa M. / Improving Students’ Practical Laboratory Techniques through Focused Instruction and Assessment / Canal, John P., Department of Chemistry, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6
Lowe, Jimmy, Department of Chemistry, British Columbia Institute of Technology, 3700 Willingdon Avenue, Burnaby, BC, Canada V5G 3H2
Fong, Rosamaria, Department of Chemistry, British Columbia Institute of Technology, 3700 Willingdon Avenue, Burnaby, BC, Canada V5G 3H2 / Chemistry Laboratory Videos: Perspectives on Design, Production, and Student Usage / Canal, John P., Department of Chemistry, Simon Fraser University, 8888 University Drive, Burnaby, British Columbia, V5A 1S6 Canada
Hanlan, Lee, Department of Chemistry, Simon Fraser University, 8888 University Drive, Burnaby, British Columbia, V5A 1S6 Canada
Key, Jessie, Department of Chemistry, Vancouver Island University, 900 Fifth Street, Nanaimo, British Columbia, V9R 5S5 Canada
Lavieri, Sophie, Department of Chemistry, Simon Fraser University, 8888 University Drive, Burnaby, British Columbia, V5A 1S6 Canada
Paskevicius, Michael, Centre for Innovation and Excellence in Learning (CIEL), Vancouver Island University, 900 Fifth Street, Nanaimo, British Columbia, V9R 5S5 Canada
Sharma, Dev, Department of Chemistry, Simon Fraser University, 8888 University Drive, Burnaby, British Columbia, V5A 1S6 Canada / Using the ACS Anchoring Concepts Content Map (ACCM) To Aid in the Evaluation and Development of ACS General Chemistry Exam Items / Reed, Jessica J., Department of Chemistry & Biochemistry, University of Wisconsin-Milwaukee, Milwaukee, Wisconsin 53211, United States
Luxford, Cynthia J., Department of Chemistry & Biochemistry, Texas State University-San Marcos, San Marcos, Texas 78666, United States
Holme, Thomas A., Department of Chemistry, Iowa State University, Ames, Iowa 50011, United States
Raker, Jeffrey R., Department of Chemistry, University of South Florida, Tampa, Florida 33620, United States
Murphy, Kristen L., Department of Chemistry & Biochemistry, University of Wisconsin-Milwaukee, Milwaukee, Wisconsin 53211, United States / How Do Chemistry Educators View Items That Test Conceptual Understanding? / Luxford, Cynthia, Department of Chemistry and Biochemistry, Texas State University, San Marcos, Texas 78666, United States
Holme, Thomas, Department of Chemistry, Iowa State University, Ames, Iowa 50011, United States / Use of the Online Version of an ACS General Chemistry Exam: Evaluation of Student Performance and Impact on the Final Exam / Elkins, Kelly M., Chemistry Department, Towson University, 8000 York Road, Towson, Maryland, 21252 United States
Murphy, Kristen L., Department of Chemistry and Biochemistry, University of Wisconsin --
Milwaukee, P.O. Box 413, Milwaukee, Wisconsin 53201, United States / Assessing the Assessments: Development of a Tool To Evaluate Assessment Items in Chemistry According to Learning Outcomes / Schmid, Siegbert, School of Chemistry, The University of Sydney, Sydney, NSW 2006, Australia
Schultz, Madeleine, School of Chemistry, Physics and Mechanical Engineering, Queensland University of Technology, Brisbane QLD 4001, Australia
Priest, Samuel J., School of Physical Sciences, The University of Adelaide, Adelaide, SA 5005, Australia
O’Brien, Glennys, School of Chemistry, University of Wollongong, Wollongong, NSW 2522, Australia
Pyke, Simon M., School of Physical Sciences, The University of Adelaide, Adelaide, SA 5005, Australia
Bridgeman, Adam, School of Chemistry, The University of Sydney, Sydney, NSW 2006, Australia
Lim, Kieran F., School of Life and Environmental Sciences, Deakin University, Melbourne, VIC 3125, Australia
Southam, Daniel C., Department of Chemistry, Curtin University, Perth, WA 6102, Australia
Bedford, Simon B., School of Chemistry, University of Wollongong, Wollongong, NSW 2522, Australia
Jamie, Ian M., Department of Chemistry and Biomolecular Sciences, Macquarie University, Sydney, NSW 2109, Australia / Editors’ Biographies /

Citation preview

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.fw001

Technology and Assessment Strategies for Improving Student Learning in Chemistry

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.fw001

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

ACS SYMPOSIUM SERIES 1235

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.fw001

Technology and Assessment Strategies for Improving Student Learning in Chemistry Madeleine Schultz, Editor Queensland University of Technology Brisbane, Australia

Siegbert Schmid, Editor The University of Sydney Sydney, Australia

Thomas Holme, Editor Iowa State University Ames, Iowa, United States

Sponsored by the ACS Division of Chemical Education

American Chemical Society, Washington, DC Distributed in print by Oxford University Press

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.fw001

Library of Congress Cataloging-in-Publication Data Names: Schultz, Madeleine, editor. | Schmid, Siegbert, editor. | Holme, Thomas A., editor. | American Chemical Society. Division of Chemical Education. Title: Technology and assessment strategies for improving student learning in chemistry / Madeleine Schultz, editor, Queensland University of Technology, Brisbane, Australia, Siegbert Schmid, editor, The University of Sydney, Sydney, Australia, Thomas Holme, editor, Iowa State University, Ames, Iowa, United States ; sponsored by the ACS Division of Chemical Education. Description: Washington, DC : American Chemical Society, [2016] | Series: ACS symposium series ; 1235 | Includes bibliographical references and index. Identifiers: LCCN 2016045121 (print) | LCCN 2016045499 (ebook) | ISBN 9780841231818 (alk. paper) | ISBN 9780841231801 Subjects: LCSH: Chemistry--Study and teaching. Classification: LCC QD40 .T435 2016 (print) | LCC QD40 (ebook) | DDC 540.71--dc23 LC record available at https://lccn.loc.gov/2016045121

The paper used in this publication meets the minimum requirements of American National Standard for Information Sciences—Permanence of Paper for Printed Library Materials, ANSI Z39.48n1984. Copyright © 2016 American Chemical Society Distributed in print by Oxford University Press All Rights Reserved. Reprographic copying beyond that permitted by Sections 107 or 108 of the U.S. Copyright Act is allowed for internal use only, provided that a per-chapter fee of $40.25 plus $0.75 per page is paid to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. Republication or reproduction for sale of pages in this book is permitted only under license from ACS. Direct these and other permission requests to ACS Copyright Office, Publications Division, 1155 16th Street, N.W., Washington, DC 20036. The citation of trade names and/or names of manufacturers in this publication is not to be construed as an endorsement or as approval by ACS of the commercial products or services referenced herein; nor should the mere reference herein to any drawing, specification, chemical process, or other data be regarded as a license or as a conveyance of any right or permission to the holder, reader, or any other person or corporation, to manufacture, reproduce, use, or sell any patented invention or copyrighted work that may in any way be related thereto. Registered names, trademarks, etc., used in this publication, even without specific indication thereof, are not to be considered unprotected by law. PRINTED IN THE UNITED STATES OF AMERICA Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.fw001

Foreword The ACS Symposium Series was first published in 1974 to provide a mechanism for publishing symposia quickly in book form. The purpose of the series is to publish timely, comprehensive books developed from the ACS sponsored symposia based on current scientific research. Occasionally, books are developed from symposia sponsored by other organizations when the topic is of keen interest to the chemistry audience. Before agreeing to publish a book, the proposed table of contents is reviewed for appropriate and comprehensive coverage and for interest to the audience. Some papers may be excluded to better focus the book; others may be added to provide comprehensiveness. When appropriate, overview or introductory chapters are added. Drafts of chapters are peer-reviewed prior to final acceptance or rejection, and manuscripts are prepared in camera-ready format. As a rule, only original research papers and original review papers are included in the volumes. Verbatim reproductions of previous published papers are not accepted.

ACS Books Department

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.pr001

Preface Although the difficulties many students encounter when learning chemistry have been known and explored for decades, there is no consensus on how best to assist and assess their learning. Over the past 10 years, the availability of a range of technological innovations that are intended to improve student learning and assessment has made the choice of teaching and assessment strategies more complex. Many teachers are rapidly adopting new technologies in teaching and assessment although their impacts have not yet been extensively studied. Piecemeal introduction of technology, widely varying contexts, and changing priorities between institutions make it difficult to draw broad conclusions about effective strategies to integrate technology into chemistry teaching. Nonetheless, many researchers have investigated the use of specific technologies in aspects of their teaching and assessment, and this book contributes to a growing body of literature that allows some generalizations to be drawn. Most importantly, specific strategies are described in detail making it possible for others to take advantage of the learning experiences and allowing practitioners to adopt the practice best suited to their needs. Some chapters also include less successful steps in the implementation of technologies, rather than an exclusive focus on a “final state” that might seem imposing to those new to the specific intervention. This book arose out of a symposium held at the 2015 Pacifichem conference in Hawaii, which in turn grew out of discussions held during Thomas Holme’s visit to Australia in 2012. The editorial team shares an interest in the assessment of learning in chemistry, which today is inextricably linked to the use of technology in teaching. Therefore, we invited submissions to the symposium to find out what others are doing in this area. The symposium itself involved 35 presentations from five countries, of which 13 have been contributed as chapters to the book. General tools for chemistry education range from tailored websites (including Web 2.0 interactive features), to optimizing the use of flipped classrooms, to the application of commercial packages in a coherent manner. The first five chapters of the book focus on these aspects of using technology directly in teaching chemistry. One area of great interest in chemistry education is the role of the teaching laboratory and how best to optimize laboratory learning. Although this was not planned as an explicit topic for the Symposium, four chapters in this book relate to different aspects of the laboratory. Two of those chapters discuss the use of short videos as instructional materials to better prepare students for the laboratory experience. One chapter relates to the use of animations for probing students’ atomic level understanding of experimental results, and the fourth explores faculty goals for laboratory learning and their relationship to students’ expectations and experiences. ix Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.pr001

The final four chapters of the book are directly related to summative assessment. Different aspects of the development and use of the multiple choice exams developed by the Examinations Institute of the American Chemical Society are described in three chapters. Finally, a metastudy describing the development of a tool to evaluate assessment items of all kinds gives the opportunity for benchmarking between institutions and even countries. The chapters in the book reflect the somewhat different teaching contexts of the countries in which the authors work. We have endeavored to provide enough information to translate between contexts while retaining the vocabulary used in each respective country. Although institutional pressures and student profiles differ somewhat, the overall goal of improving teaching of undergraduate chemistry is shared. Because the specific details of the learning challenges being addressed tend to vary slightly between countries, the solutions that are found are similarly diverse. The result of this mixture of challenges being met is that the resources described in this book are capable of seeding new ideas in each of the environments from which the chapters are drawn. The chapters differ in their scale, ranging from local applications of technology tools to larger national studies, but all describe results that can be applied more broadly. We hope that the reader will find the content interesting and useful in their teaching.

Technology Tools for Chemistry Education 1.

Lawrie, Schultz, Bailey, Al Mamun, Micallef, Williams, and Wright present a study in their chapter that describes how online modules can be constructed to assist the growth of conceptual understanding for students in university-level chemistry classes. In particular, they note three elements that should be present to assist students who are using online resources: (1) scaffolding of the learning experience; (2) visual representations at multiple scales; and (3) routine feedback as students progress through the modules. Patterns of student usage of the modules support the conclusion that this development strategy produces learning tools that students find helpful.

2.

In the chapter by McCollum, the challenge of maintaining traditional skill sets for students while incorporating new methods is described. Specifically, with newer teaching methods, including the use of flipped classrooms which prompt students to engage with materials on their own prior to in-class activities, the ability of students to gain information from reading has become increasingly important. Not all students, however, have sufficiently honed scientific reading skills, so building academic reading circles and assigning students specific roles to fill in these activities can improve learning outcomes when new reading-centric pedagogies are implemented.

3.

In her chapter, Lawrie emphasizes the need to incorporate a multidimensional strategy for teaching with new technologies. This x

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.pr001

model incorporates the skills, knowledge, and experiences of students as well as teachers. By using practices that have been identified via research as being important for student success, the introduction of new web-based learning tools can be implemented and assessed so that learning outcomes are enhanced. Several examples of how this strategy has been employed are included to illustrate the practical implementation needs for employing new technologies in teaching. 4.

Redd, Gravely, Lewis, and Redd in their chapter describe the integration of a set of technology tools to improve the student learning experience. While learning management systems (LMSs) are widely used, they are rarely integrated into the classroom efficiently. Lesser known packages fill the gaps in the LMS, allowing a coherent digital platform that offers differentiated teaching and ease of use.

5.

Venkateswaran provides in her chapter an exemplar of how new learning technologies that are emerging among textbooks can influence student learning. Increasingly publishers are packaging a range of tools as part of the student textbook materials, and the onus lies with instructors to find ways to leverage the new diversity of tools. This chapter describes the trajectory taken by an instructor for the introduction of adaptive learning tools associated with a textbook, including both textual and homework support.

Laboratory Learning 6.

Bretz, Galloway, Orzel, and Gross investigated the connection between faculty goals for learning in the undergraduate general chemistry and organic chemistry laboratory, the experiments conducted in these labs and students’ expectations, and experiences with regard to meaningful learning. Data were collected for all three aspects and the analysis showed that faculty goals do not always align with the selected experiments and that there is little connection between faculty goals and students’ learning.

7.

Kelly designed a number of animations depicting a certain experimental procedure. The animations were designed with significant differences, and students were asked to critique them and select the one that best portrayed the experiment. This exercise provides students the opportunity to practice critiquing the plausibility of animations as they fit with experimental evidence. In a follow-up justification exercise, it was observed that many students were challenged when they had to articulate why the animations’ features fit with the experimental evidence. The justification exercise also revealed that students need more practice learning to critique models in connection to experimental evidence, and how to write in a manner that conveys their thought process. xi

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.pr001

8.

Canal, Lowe, and Fong have reevaluated and modified the way instruction on laboratory techniques is delivered to students in their laboratory courses. In order to focus students’ attention on the proper way to use the glassware and common apparatuses used in most undergraduate laboratories, a laboratory-techniques experiment was developed as well as laboratory technique-centred exercises. They present their approach to improve student learning, instructor observations, and data to support the effectiveness of these initiatives.

9.

Canal, Hanlan, Key, Laveiri, Paskevicius, and Sharma investigated the effectiveness of instructional videos as a teaching tool in the chemistry laboratory curricula at both Simon Fraser University (SFU) and Vancouver Island University (VIU). Five categories of videos used in first, second, and third year laboratory courses were developed, either in-house (by faculty) or with the assistance of visual media professionals. Short student feedback surveys from both institutions indicate that students find the videos to be an effective tool in their education. Most students felt they were better prepared and more confident about their experiments after watching the videos.

Evaluating Summative Assessment 10. Murphy and her colleagues have investigated the use of a comprehensive list of chemistry concepts (the Anchoring Concepts Content Map) to categorize multiple choice exam questions. Use of the list can highlight topics that are not included in the exam and can also aid the preparation of new exams. In this chapter the list is applied to the ACS Exams Institute exams, but it can also be applied to any exam. 11. Luxford and Holme have surveyed a large number of chemistry educators in the US about their views of conceptual understanding in general chemistry and how it can be assessed. In their chapter, they analyze the part of their survey in which participants were asked whether six different mock items test conceptual understanding. The outcomes were correlated with the participants’ personal definitions of what conceptual understanding is. The results are interesting in the diversity of responses to some mock items and may provide insight to educators attempting to frame their own conceptual multiple-choice items. 12. Elkins and Murphy describe in their chapter the use of an online version of an ACS exam as an optional practice exam for low stakes in preparation for a final exam. Students who took the ACS exam for practice performed somewhat better on their final exam although they were not told their specific areas of difficulty. Also, it was found that weaker students performed as well as stronger students in questions on experimental work and involving visualization. xii Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.pr001

13. A large-scale collaboration is described by Schmid, Schultz, Priest, O’Brien, Pyke, Bridgeman, Lim, Southam, Bedford, and Jamie. This work is associated with the implementation of Chemistry Threshold Learning Outcomes (CTLOs) in response to regulatory demands in Australia. Finding a methodology to evaluate assessments required an iterative approach and resulted in rubrics that provide insight into how departments view their own efforts to assess the CTLOs. In particular, these local efforts tend to over emphasize the extent to which assessments measure student achievement related to the CTLOs. By providing a template for deep analysis of assessment materials, the overall goal of improving student learning can be better advanced in parallel with satisfying regulatory requirements. The tool can be applied to any set of learning outcomes. We are grateful to all of the reviewers who took the time to carefully read and review the chapters of this book. We also thank the ACS Examinations Institute for supporting the Pacifichem Symposium. The authors also acknowledge support from the Australian Government’s Office for Learning and Teaching (grant OLT ID14-3652).

Madeleine Schultz Queensland University of Technology Brisbane, Australia [email protected] (e-mail)

Siegbert Schmid The University of Sydney Sydney, Australia [email protected] (e-mail)

Thomas Holme Iowa State University Ames, Iowa, United States [email protected] (e-mail)

xiii Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Chapter 1

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

Development of Scaffolded Online Modules To Support Self-Regulated Learning in Chemistry Concepts Gwendolyn A. Lawrie,*,1 Madeleine Schultz,2 Chantal H. Bailey,3 Md. Abdullah Al Mamun,4 Aaron S. Micallef,5 Mark Williams,6 and Anthony H. Wright7 1School

of Chemistry & Molecular Biosciences, The University of Queensland, St Lucia, QLD 4072, Australia 2School of Chemistry, Physics and Mechanical Engineering, Queensland University of Technology, Brisbane, QLD 4001, Australia 3School of Chemistry & Molecular Biosciences, The University of Queensland, St Lucia, QLD 4072, Australia 4School of Education, The University of Queensland, St Lucia, QLD 4072, Australia 5School of Chemistry, Physics and Mechanical Engineering, Queensland University of Technology, Brisbane, QLD 4001, Australia 6School of Science & Health, Western Sydney University, Penrith, 2751 NSW, Australia 7School of Education, The University of Queensland, St Lucia, QLD 4072, Australia *E-mail: [email protected]

Teachers rely on representations, simulations and animations in their classrooms to explore and expand students’ conceptual understanding in chemistry. Researchers adopt the same visualization tools to investigate student understanding and to support their communication of the outcomes of their research studies. In the past decade, many carefully designed web-based resources including sophisticated simulations and animations have been developed and are accessible online for teachers to engage students in guided and inquiry activities. In spite of decades of research on student difficulties with conceptual understanding, there are few examples of modules

© 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

incorporating these online resources designed to improve students’ understanding of concepts that underpin learning in tertiary chemistry. In this project, the design and delivery of five online modules covering fundamental chemistry concepts has been explored, informed by research literature in the areas of scaffolding and visual representations. The aim was to encourage students to engage in self-regulated exploration of these modules, initiated by the provision of formative feedback through a diagnostic instrument. Two separate mechanisms for delivering online modules, both integrating existing web-based resources, were trialed and evaluated in terms of student engagement and perceptions.

Introduction Multimodal representations of models, processes and concepts at the macroscopic, submicroscopic and symbolic levels are integral to learning chemistry. To engage with these representations, students must develop skills in translation between the various forms, as well as assigning meaning to them. Many excellent online resources for teaching chemistry concepts using these representations are readily accessible including interactive simulations such as Molecular Workbench (1, 2) and PhET (3, 4). These sophisticated resources are designed as tools that teachers can integrate into their teaching contexts with multiple shared lesson plans and exemplars for practice on the respective websites. The tools have also now been applied in research studies seeking to understand whether online dynamic visualizations support student learning and related factors (5–8). While teachers adopt these resources to support their students in the construction of understanding (9), there are few studies that report the development of interventions aiming to correct misconceptions (10–12). Improving the understanding of basic chemistry concepts is somewhat of a holy grail in chemistry education. Extensive research over many decades shows that alternative conceptions are often established in the early years (13–15) and can be very persistent (16, 17). Instruments profiling misconceptions in many different sub discipline areas of chemistry have been published (18–21). Remediation of misconceptions may require existing understandings to be challenged through cognitive conflict and then rebuilt (22–25). However, if a consistent world view does not exist in the student, there may be a lack of conception rather than a misconception and this should be easier to remedy (17, 26). In this project, we aimed to design and implement self-regulated online learning modules that support the construction of conceptual understanding in chemistry within five topics. The stand-alone modules were developed to be suitable for independent use by incoming tertiary students to enable them to direct their own learning (27) and remediate missing and missed conceptions (28). A review of the literature indicates that the following components should be incorporated in the design (29–31): 2 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

• • •

Scaffolding (32); Representations (33); and Formative feedback (34).

Monitoring factors that affect cognitive load is also important in the design phase in order to avoid unnecessary overload on users (35, 36). A critical issue in development of the modules was the level of scaffolding required for inquiry learning to be effective in the context of the simulations, animations and representations. The concept of scaffolding was developed in the context of teacher- (or parent-) student interactions, and original definitions require it to be dynamic, fading as students become more competent (37). Several authors have argued that without additional one-on-one scaffolding provided by a teacher, computer-based scaffolding is ineffective (38, 39). It remains an open question as to whether computer-based scaffolds need to exhibit dynamic assessment to be termed scaffolds. Within the scaffolding framework, dynamic assessment is intended to help the teacher provide just the right amount of support at just the right time to students. Several potential dangers of not dynamically adjusting scaffolding support have been noted. First, some authors caution that by failing to dynamically adjust support, designers may fail to promote students’ ability to independently perform the supported task... Second, some authors note that failure to dynamically assess student ability may cause cognitive overload on the part of students who can already accomplish portions of the task effectively... (reference (37) p. 513) However, scaffolding strategies to support online inquiry learning have been proposed (40, 41). These guidelines include (a) explicitly describing the structure of online inquiry tasks via visual representations so learners can better understand tasks they may only naively understand; (b) incorporating planning tools so that learners can think about their tasks in advance and plan their online inquiry more often; (c) making the online inquiry process, the working history through that process, and information common to multiple activities explicit to learners so they can monitor and regulate their work; and (d) providing reflection support through prompts to help learners see what they should reflect on and articulate throughout their online inquiry. (ref. (40), p. 242) These approaches allow students to effectively self-scaffold by skipping repetition when they are confident that they understand a section. An example of a computer tutorial that uses scaffolding effectively in teaching students organic chemistry has been published (42). Multimedia (33) and animations (43) have been shown to assist students as they begin to transition between the macroscopic, sub-microscopic and symbolic representations used in chemistry (44). Tasker has developed a suite of materials 3 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

known as VisChem that has been recognized as best practice in learning design to enable students to visualize chemistry using all three representations (45–47). The VisChem methodology of representing molecules graphically was adopted throughout the project. Concept check questions were included regularly within the modules to assist students to monitor their learning through immediate formative feedback (34). The importance of immediate feedback has been emphasized as critical to improving understanding (29). A model for IT continuance has been established that integrates factors likely to lead people to use technology again after they have tried it (48). Designing and building a website is time consuming and expensive so it is critical to be aware of factors that are likely to lead to its adoption and continuance. Awareness of these factors informed our approach as the project evolved. This project formed part of a broader project that began by profiling the conceptions of incoming tertiary students at five universities in Australia (49) with the aim of providing formative feedback and remediating misconceptions. The methodology of the project was to use student responses to clusters of questions on chemistry topics covering multiple concepts to direct them to online activities tailored to their specific difficulties. Having established an instrument for this purpose (50) the present manuscript describes the process of development and implementation of the online modules, and outcomes from the first two years of its use. Affordances and limitations of the different strategies are described to provide recommendations for practice.

Design Methodology Design of the online modules was guided by the need to deliver tailored activities appropriate for each student’s current level of understanding. Students were directed to suitable activities according to the combination of responses that they selected in the ordered multiple-choice items (51, 52) of the diagnostic instrument (50). Thus, not only whether they were correct, but also their choice of distractors was relevant to the selection of suitable activities. In order to cover the range of student understanding in large, diverse first year chemistry cohorts, four categories were developed with different objectives as shown in Table 1. Further feedback provided after students’ initial interactions with the online modules moved them to other categories and towards improved conceptual understanding. This iterative cycle (Figure 1) was designed to support students in the transition into tertiary chemistry studies (27, 49). A storyboard was established for each of the five topics to elaborate the combination of activities, instructions and elements that would be incorporated for students within each category. Within each topic covered in the diagnostic instrument, multiple concepts are required for understanding, so the activities were organized around the headings in Table 2. These headings were not determined in advanced but grew organically from an analysis of distractor choice in the instrument and misconceptions common to many students. Figure 2 shows a screenshot of part of the storyboard for one of the topics. 4 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

Table 1. Categories for the structure of online activities Exercise category

Students level of understanding

Online activities

Concept Builder

Student’s responses indicate negligible understanding of concept.

Introduce student to the concepts through foundation ideas.

Concept Fix

Student possesses a significant alternate conception.

Introduce cognitive dissonance to challenge student’s conceptions.

Concept Shift

Student possesses a minor alternate conception.

Present student with an alternative model to clarify concept.

Concept Quest

Student possesses well-formed conceptions.

Provide student with the opportunity to apply and extend their understanding.

Figure 1. Iterative cycle of feedback and activities. (Adapted from Ref. (49) under Commons Attribution Licence (CC-BY). Copyright 2013).

In addition to careful storyboarding, the design of each module required careful consideration of the learning objects that were sourced. We integrated activities from existing high quality online resources suitable for the topics that we addressed (2, 3, 46). YouTube videos were included where directly relevant. Best practice in representing chemical processes, particularly in aqueous solutions, within graphics and animations from the VisChem project (45–47) was adopted. The different visualization tools that embedded in the modules can be classified in the following categories (53): • • • •

simulation - an interactive dynamic representation that is pictorial; animation - a dynamic representation that is pictorial; video - a dynamic visualization that is photorealistic; static diagram - a graphical representation that relies on some abstraction. 5

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

Table 2. List of topics and subtopics covered by the diagnostic instrument item clusters and online modules Topic

Subtopics / concepts

Phase change

intermolecular interactions molecular water states of matter

Conservation of matter

balancing equations fate of matter reaction stoichiometry

Aqueous solutions

dissolution speciation dissolving salts and solubility dissolving salt proportional reasoning

Heat and energy

heat transfer thermal expansion making bonds energy and reactions

Chemical equilibria

chemical equilibria dynamic equilibrium Le Châtelier’s principle saturated solutions

Figure 2. Part of storyboard overview for the Phase Change module activities. (Reproduced with permissions from Refs. (2), (3), and (46). Copyright 2013 and 2016). 6 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

It has been shown that including narration with visual presentation reduces cognitive load and may assist student understanding (54). In our teaching practice we have noticed that many students need some demonstration of how to use simulations in order to understand what they do and how they can be controlled and explored. We wanted to guide students to particular activities and in some cases restrict the complexity of the simulations to focus attention on a particular phenomenon. Thus tutorials were created where simulations were implemented in the modules in both structured and explorative manners, so that students received different levels of guidance in how to interact with the simulations. For example, in the module on chemical equilibrium, some PhET simulations were carried out by the researchers and included in the modules as animations with embedded instructions to assist students to later carry out the simulation as envisioned for the activity. A screen shot of such an animation tutorial is shown in Figure 3.

Figure 3. Screen shot from a video scaffolding students’ use of a PhET simulation in ChemBytes Chemical Equilibrium module highlighting important interactive features. (Reproduced from Ref. (3) under Commons Attribution Licence (CC-BY). Copyright 2016. https://shire.science.uq.edu.au/chembytes/Index.html#/Home/Main).

Based on their experiences in implementing these simulations in their classrooms, project team members decided that guided instruction is more useful than open investigation when using sophisticated simulations with many variables. Another form of scaffolding is posing questions to direct student interaction with the simulation. This initiates students’ exploration and later on guides them to open inquiry. 7 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

A website was built around the storyboard for each module with progression through the categories listed in Table 1 and incorporating the different visualization tools. The modules were designed to be completed asynchronously (in self-directed manner) or synchronously as part of classroom activities as a form of ‘blended’ learning. In the first iteration, known as ReSOLv (illustrated by element symbols), students were sent an individual password to access the website (detailed in evaluation methodology below) by email. This email also included their score on the diagnostic instrument and the class performance for each question as a percentage of students who answered correctly. Based on their performance in the diagnostic instrument students were directed to a specific starting point in one of the categories listed in Table 1. Analytics were used to track student engagement and progress through the various web pages and activities. It should be noted that engagement with the website was not required as part of any teaching activities or assessment at any of the participating universities in semester 1, 2013. In this iteration of the website, a student’s progress through the activities was dependent on their entry level. This was structured through a flowchart for each module. An example is shown in Figure 4. The path included concept check questions for immediate feedback after each activity.

Figure 4. Flowchart for the structure of the ReSOLv website showing the scaffolding involved in students’ progress through the Heat and Energy module.

This structure led to low rates of usage by students and so an open website was developed. In particular, the flowchart forces the students’ order of engagement with the material and was abandoned. 8 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

The second platform trialed was ChemBytes, a series of bespoke web pages. The design of these web pages was inspired by the Five Minute Physics project from the University of Queensland (55, 56) which is designed to engage students in concepts and topics in physics courses as part of a flipped classroom delivery. The features that appealed to the project team included the app-like icons for different topics, the ‘How am I doing’ check questions and the ‘Summing up’ reprises. We adopted the scaffolding approached advocated by Quintana for online learning (40), explicitly describing the tasks and allowing students to plan, monitor and reflect on their progress. A pilot platform elicited insight into student engagement and informed additional desirable design features such as additional white space, less text, more images (macroscopic and submicroscopic representations) and the options for students to interact with animations and simulations. The design minimized unnecessary cognitive load (36) and explicitly made connections between representations as recommended (33). Figure 5 shows a screenshot of a page within ChemBytes to illustrate the look and feel that was achieved. The ChemBytes pages can be accessed from the project website http://www.iammicproject.com.

Figure 5. Screenshot of a landing page within the topic of Phase Change in ChemBytes. (Reproduced with permissions from: https://shire.science.uq.edu.au/chembytes/Index.html#/Home/Main)

Students were directed to the ChemBytes web pages through URLs embedded within their institutional LMS. Google Analytics was enabled to collect data with regard to student engagement with the web pages, including the times when students accessed the site, the average time spent on the site, location from which they accessed the site, devices they used and browsers. The modules were recommended by the instructors in general chemistry courses at two institutions in semester 1, 2014 as part of students’ self-directed studies. 9 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

Evaluation Methodology This study is a subcomponent of a larger project which has investigated the provision of formative feedback in relation to students conceptions in chemistry in order to direct them towards online learning modules. The larger study implemented the Learning Environment, Learning Process and Learning Outcomes (LEPO) evaluation framework (57) to collect and analyse data. This evaluation was supported by ethical clearance secured in all five participating universities. Students who participated in the current study were first-year general chemistry students who were enrolled across five Australian universities, situated in three Australian states. They were enrolled in a diverse set of programs of study including, but not limited to, engineering, biotechnology, materials science, agricultural science, medicine, pharmacy, dentistry, biomedical science and health sciences. Student activity in the ReSOLv web platform was monitored through web analytics for individual students. Students were initially sent a login password as part of their feedback from a diagnostic concept diagnostic instrument (50). ChemBytes was designed to replace ReSOLv after early evaluation of ReSOLv revealed its low uptake so the two platforms were subjected to a parallel comparison. The evaluation of ChemBytes included analytics data collected through the website and statistics available through the learning management systems, Blackboard (Bb) and Moodle. Participant recruitment was through email invitation sent to the whole class enrolment and informed consent, with a provision to opt out of the study at any time, for both the online questionnaires and focus group interviews. The online questionnaire was delivered at the end of semester 1 in only one university where the online modules were provided through Bb as optional study resources prior to completion of a summative quiz. The questionnaire contained several quantitative scales exploring the learning environment, student motivation and the data for only one item presented here. As part of a cluster of items evaluating learning in activities in the course, the following question was asked: How much did completing the CROM online modules help your learning? Scale answer options included ‘No help’, ‘A little help’, ‘moderate help’, ‘much help’ and ‘great help’. Several open questions in the online questionnaire explored students’ perceptions in relation to the online modules delivered through Chembytes and other online resources. While this questionnaire captured a range of data related to multiple course activities, the specific open response question that was of interest to this study was framed to ilicit a range of responses: ‘Many of the [modules] used interactive visualizations / animations / videos of molecular level processes. Which of these were the most useful for helping you build your understanding in chemistry?’ 1536 students were enrolled in the course and were invited to complete the online questionnaire through an email sent through Bb, and 1003 students responded to this question. This open response data was thematically coded by an inductive process in NVivo to identify emergent themes that were then used as categories. Two separate focus group interviews were conducted to explore the accessibility and barriers to the use of the two different platforms 10 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

for delivery of the modules. Interview 1 involved 7 participants and interview 2 involved 8 participants (N = 15 total).

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

Results and Discussion Three key elements were incorporated into the design of the online modules: scaffolding, visual representations and feedback. During the original online module design, the project team intended that students would be required to login to the online activities so that their progress could be guided and monitored. However, several disadvantages were encountered in the pilot of the activities on the ReSOLv website which indicated that student entry and progression in the modules was over-scaffolded. Firstly, generating and distributing individual login information for thousands of students enrolled in multiple universities became complex and time consuming. Secondly, students were not observed to engage significantly with the system, presumably partly because of the requirement to login with details that were sent to them rather than self-initiated. 1654 students enrolled in four separate Australian universities were sent a login URL and password to access the ReSOLv modules. 141 (8.5%) students logged into ReSOLv and 86 students did not continue despite 5 of these logging in on two occasions. Of the students that did complete activities, it is not possible to discriminate the 3.3% students by institution because 19 of this group supplied a personal contact email (@gmail, @hotmail, @live, @yahoo etc) rather than their student account. The ReSOLv website had highly structured pathways guiding students through the activities (Figure 4). Progress was scaffolded and there was limited opportunity for students to iterate activities at will. In contrast, ChemBytes did not require a dedicated login and students were provided access through a hyperlink to a website delivered through the learning management system. Each instructor was able to deliver the online modules to fit their context and to align with their curriculum’s learning progressions. Figure 6 presents a schematic flow chart for how students move through a module comparing ReSOLv with ChemBytes. The adoption of a website to deliver online modules reduces the opportunities to track student engagement with learning objects as individual ‘clicks’ made by a single student are no longer accessible. Ideally, the usability of ChemBytes is best explored through observation and interviews with a small number of students as they engage with the learning activities. In this study, we were evaluating over a thousand students and evaluation is restricted to Google Analytics and students self-reported perceptions and feedback in online questionnaires and focus groups. Google Analytics data collected during one semester for the use of ChemBytes over four of the modules is presented in Table 3. It can be seen that although the number of users drops slightly as the semester progresses, the average time per session remains above 10 minutes and around half of the approximately 1500 students in this group used each module of ChemBytes.

11 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

Figure 6. Flow charts showing typical student progress through ReSOLv (upper chart) compared with ChemBytes (lower chart) modules. (Reproduced with permissions from: https://uwssites.uws.edu.au/equiz/iammic/login.php and https://shire.science.uq.edu.au/chembytes/Index.html#/Home/Main). A limitation of Google Analytics in the evaluation of student activity on website pages is that it cannot supply individual information in regard to the identity of users so it was not possible to monitor individual students. To overcome this, identifying information could be collected by embedding a text field directly in the web page; however, our aim was to encourage self-regulation in our students through their use of these study resources. The project team believed that maintaining their anonymity would better support their independent exploration of these web resources. Google Analytics does provide useful 12 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

information in terms of demographic information for all users including: gender, location, device, operating system and browser. In alignment with the IT continuance model, although the core content of the modules did not change substantially, moving from the ReSOLv website to ChemBytes improved multiple factors including “facilitating conditions” (no login required) and “satisfaction” (faster) that are likely to lead to continuance behavior. Barriers to access to ReSOLv raised by students in interviews included login difficulties, Java/Flash problems and problems using the site on mobile devices; all of these fall under facilitating conditions for continuance (48). While learning online, students prefer all content to be run in the same window. During interviews, many students reported that the PhET simulation (Java applet) opened in a different window and this appeared to have impacted on their engagement.

Table 3. ChemBytes Google Analytics data for use of ChemBytes, semester 1, 2014 Module

Number of Sessions

Number of Users

Page Views

Average time

Phase Change

1441

1092

5011

11 min 17 sec

Heat & Energy

1361

999

5449

12 min 25 sec

Equilibrium

1108

832

4290

15 min 22 sec

Aqueous Solutions

785

653

2616

10 min 21 sec

One of the design elements was the deliberate incorporation of representations to support learning. Inductive thematic coding of student responses to the survey asking which visualization mode was most useful distilled four major themes: • • • •

‘seeing’ molecules; explanations (audiovisual); interactive; and “visualizations did not help”.

Figure 7 provides the breakdown of responses into each category. It should be noted that students were not provided with definitions for each category and so there was substantial crossover and substitution of terminology in their responses, in particular between animation and video. A total of 1003 enrolled students responded and consented to participate in the study (65% completion rate) to the survey. While there was not a specific question that asked these students whether they had used ChemBytes, a five point quantitative scale item indicated that 3.8% students had found the online activities were of ‘no help’ in their learning and 67.4% found them to be ‘much help’ or ‘great help’. While there is no doubt that students enjoy multimodal representations, students identified with ‘seeing’ molecular level structures and processes to the greatest extent.

13 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

Figure 7. Percentage of responses to open response items on the student questionnaire coded into each theme.

Table 4 provides example quotes from student responses to the open ended items in regard to ChemBytes (each quote is chosen to be representative of at least 10 similar quotes). Many students referred to their preferred way of learning in terms of being a ‘visual’ learner or not and it was clear that a combination of modalities (multi-modal) is ideal with several students referring to the complementary nature of the combination of representations (Table 4). During the process of designing the modules, audio explanations were not considered as a required element - it was hoped that students might be encouraged to explore the interactive elements rather than be directed by explanation. It was evident that a significant number of students preferred audio explanations of concepts through the videos or animations rather than the option of self-directed exploration (Table 4). This is consistent with research findings that narration combined with visualization reduces cognitive load (54). Approximately 20% of responding students indicated that they enjoyed interactive visualizations such as PhET and Molecular Workbench. It was clear from their comments that the ability to decide on the variable (control) was a critical factor.

14 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

Table 4. Examples of typical student responses regarding the format of representation that was most useful in supporting their learning (online questionnaire) Representational feature

Example student perception

‘Seeing’ molecular level phenomena

From the videos and animation, we can see the molecules moving and reacting in a chemical reaction clearly which help me visualize the whole process

Audio explanations

The videos were most helpful because someone else was able to explain to me the processes that were occurring. A lot of the time, the animations and interactive visualizations did not explain what was going on, so were useless.

Interactive exploration

Interactive visualizations were the most helpful for me, because I was able to understand the concept more clearly due to the fact that I could control it and see the differences that occurred when I changed a particular aspect.

Visual learner

I think that the visualizations were always useful as I am more of a visual learner, and if I am able to see images of processes, I find that it becomes a lot clearer. Also, when I need to recall on these processes later on, I find that I can just think about the images in my head

Non-visual learner

I didn’t find the visualizations useful, i prefer to learn through explinations [sic] rather than visually

Focus on calculations

none, the difficulty in the course is entirely based on the math, not visual concepts

Multimodal

All of them were somewhat useful by providing a unique way of learning chemistry. I don’t consider one to be better than the other though (they complimented [sic] each other)

One of the aims in the design of the modules was to incorporate elements that would appeal to the majority of students. However, 12% of those that responded indicated that they had not found the visualization activities useful. These students were divided in their reasons, which included: • • •

they were already familiar with concepts so did not feel they needed to access resources; they indicated that they were not visual learners and preferred to read text instead; and they preferred to learn through rehearsing or solving calculations.

The most troubling feedback was the group of students (3%) that claimed computer issues had prevented them from accessing various resources. They had not sought assistance that was widely available. This emergent issue of access and student technological skills was also apparent in focus group feedback, several students identified issues with their browsers and Java in particular, for example: 15 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

I use Safari a lot because I’ve got a Mac and I found that the activities didn’t like being used with Safari. You had to use them Mozilla Firefox instead and because I’m always using Safari I’d have to close down Safari then log in on Mozilla, watch the activities and then do that and I’d have to do it every time because I’m always in Safari, that’s just what I’ve got open, that’s what it is (Focus Group 1) It wanted you to download stuff onto your computer. And, like, my computer’s a little bit touchy sometimes, like, there was one which I just didn’t download because I was concerned it was going to have a virus in it or something because my computer started, like, flashing alarm things at me. Yeah, but I found the technical difficulties with the browsers and downloading stuff to be a bit frustrating. (Focus Group 2) This finding is important in terms of widening student access to web-based learning modules. It is easy to assume that students have high levels of digital literacy. However, for our study it was clear that this was not the case and additional scaffolding or support is required to assist students to manage access through their own devices. While we have no direct data from this study in regard to individual students’ shifts in conceptual understanding through use of the online modules, the attitudes and perceptions described indicate that students mostly found these web-based modules to be useful. It can be inferred that students perceive usefulness through their continuance with the technology. The factor of post-usage usefulness is critical in determining continuance intention (48).

Conclusions Five engaging online modules for addressing conceptual weaknesses of undergraduate students have been developed incorporating scaffolding, representations and feedback. Best practice in representations for chemistry was adopted to minimize potential cognitive load and enable visualization on the macroscopic and submicroscopic scales. This practice was informed by research (36) and the project team’s own expertise (45–47, 58). Two online web platforms were trialled to deliver these carefully designed modules and greater engagement of students was achieved in the open access website ChemBytes. Scaffolding and formative feedback have been incorporated into the modules to allow student monitoring of and reflection upon their progress. Student adoption and continuance with the modules reflects the manner in which they have been integrated by teachers at their universities. Feedback in focus groups and surveys together with data showing high numbers of returning users suggest that students find the modules useful. The use of an open website or a single login with institutional credentials is critical to student engagement.

16 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Acknowledgments We thank all our student participants in focus groups and surveys and those who tested the websites. We thank the members of the project team Simon Bedford, Tim Dargaville, Glennys O’Brien, Roy Tasker and Chris Thompson for their creative input into module designs. We are also grateful to Marnie Holt, Trevor Daniels and Tanya Brady, who formed the technical team that developed the graphics and created the ChemBytes website. The ReSOLv website was created by one of the authorship team (Williams).

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

References 1.

Xie, C.; Tinker, R.; Tinker, B.; Pallant, A.; Damelin, D.; Berenfeld, B. Computational experiments for science education. Science 2011, 332, 1516–1517, DOI: 10.1126/science.1197314. 2. The Concord Consortium. Molecular Workbench. http://mw.concord.org/ modeler/ (accessed June 17, 2016). 3. PhET Interactive Simulations. PhET. https://phet.colorado.edu/(accessed June 17, 2016) . Creative Commons Attribution Licence (CC-BY) link: https://creativecommons.org/licenses/by/3.0/at/deed.en_GB. 4. Wieman, C. E.; Adams, W. K.; Perkins, K. K. PhET: Simulations that enhance learning. Science 2008, 322, 682–683, DOI: 10.1126/science.1161948. 5. Moore, E. B.; Herzog, T. A.; Perkins, K. K. Interactive simulations as implicit support for guided-inquiry. Chem. Educ. Res. Pract. 2013, 14, 257–268, DOI: 10.1039/C3RP20157K. 6. Khoshouie, E.; Ayub, A. F. M.; Mesrinejad, F. Molecular workbench software as computer assisted instruction to aid the learning of chemistry. J. Educ. Soc. Res. 2014, 4, 373–379, DOI: 10.5901/jesr.2014.v4n3p373. 7. Russell, J.; Kozma, R. B. Assessing learning from the use of multimedia chemical visualiztion software. In Visualization in Science Education; Gilbert, J. K., Ed.; Springer: Dordrecht, Netherlands, 2005; pp 299−332. 8. Feldman-Maggor, Y.; Rom, A.; Tuvi-Arad, I. Integration of open educational resources in undergraduate chemistry teaching – a mapping tool and lecturers’ considerations. Chem. Educ. Res. Pract. 2016, 17, 283–295, DOI: 10.1039/c5rp00184f. 9. Rutten, N.; van Joolingen, W. R.; van der Veen, J. T. The learning effects of computer simulations in science education. Comput. Educ. 2012, 58, 136–153, DOI: 10.1016/j.compedu.2011.07.017. 10. Regan, Á.; Childs, P.; Hayes, S. The use of an intervention programme to improve undergraduate students’ chemical knowledge and address their misconceptions. Chem. Educ. Res. Pract. 2011, 12, 219–227, DOI: 10.1039/C1RP90027G. 11. Treagust, D. F.; Chandrasegaran, A. L.; Zain, A. N. M.; Ong, E. T.; Karpudewan, M.; Halim, L. Evaluation of an intervention instructional program to facilitate understanding of basic particle concepts among students enrolled in several levels of study. Chem. Educ. Res. Pract. 2011, 12, 251–261, DOI: 10.1039/C1RP90030G. 17 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

12. Garnett, P.; Hackling, M. W.; Oliver, R. Development of an interactive multimedia package designed to improve students’ understanding of chemical equations. Paper presented at 21st Annual Conference of the Western Australian Science Education Association, Perth, 1996. 13. Osborne, R. J.; Cosgrove, M. M. Children’s conceptions of the changes of state of water. J. Res. Sci. Teach. 1983, 20, 825–838, DOI: 10.1002/ tea.3660200905. 14. Mitchell, I.; Gunstone, R. Some student conceptions brought to the study of stoichiometry. Res. Sci. Ed. 1984, 14, 78–88, DOI: 10.1007/BF02356793. 15. Peterson, R. F.; Treagust, D. F.; Garnett, P. Development and application of a diagnostic instrument to evaluate grade 11 and 12 students’ concepts of covalent bonding and structure following a course of instruction. J. Res. Sci. Teach. 1989, 26, 301–314, DOI: 10.1002/tea.3660260404. 16. Taber, K. S.; Tan, L. C. D. The insidious nature of ‘hard core’ alternative conceptions: Implications for the constructivist research programme of patterns in high school students’ and pre-service teachers’ thinking about ionisation energy. Int. J. Sci. Ed. 2011, 33, 259–297, DOI: 10.1080/09500691003709880. 17. Cooper, M.; Corley, L. M.; Underwood, S. M. An investigation of college chemistry students’ understanding of structure–property relationships. J. Res. Sci. Teach. 2013, 50, 699–721, DOI: 10.1002/tea.21093. 18. Brandriet, A. R.; Bretz, S. L. The development of the Redox Concept Inventory as a measure of students’ symbolic and particulate redox understandings and confidence. J. Chem. Educ. 2014, 91, 1132–1144, DOI: 10.1021/ed500051n. 19. Luxford, C. J.; Bretz, S. L. Development of the Bonding Representations Inventory to identify student misconceptions about covalent and ionic bonding representations. J. Chem. Educ. 2014, 91, 312–320, DOI: 10.1021/ed400700q. 20. Wren, D.; Barbera, J. Gathering evidence for validity during the design, development, and qualitative evaluation of thermochemistry concept inventory items. J. Chem. Educ. 2013, 90, 1590–1601, DOI: 10.1021/ed400384g. 21. Mulford, D. R.; Robinson, W. R. An inventory for alternate conceptions among first-semester general chemistry students. J. Chem. Educ. 2002, 79, 739–744, DOI: 10.1021/ed079p739. 22. Hand, B. M.; Treagust, D. F. Application of a conceptual conflict teaching strategy to enhance student learning of acids and bases. Res. Sci. Ed. 1988, 18, 53–63, DOI: 10.1007/BF02356580. 23. Li, S. C.; Law, N.; Liu, K. F. A. Cognitive perturbation through dynamic modelling: A pedagogical approach to conceptual change in science. J. Comp. Assist. Learn. 2006, 22, 405–422, DOI: 10.1111/j.1365-2729.2006.00187.x. 24. Limón, M. On the cognitive conflict as an instructional strategy for conceptual change: A critical appraisal. Learn. Instr. 2001, 11, 357–380, DOI: 10.1016/S0959-4752(00)00037-2. 18 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

25. Clark, D.; Jorde, D. Helping students revise disruptive experientially supported ideas about thermodynamics: Computer visualizations and tactile models. J. Res. Sci. Teach. 2004, 41, 1–23, DOI: 10.1002/tea.10097. 26. Kirbulut, Z. D.; Beeth, M. E. Consistency of students’ ideas across evaporation, condensation, and boiling. Res. Sci. Ed. 2013, 43, 209–232, DOI: 10.1007/s11165-011-9264-z. 27. Nicol, D. Assessment for learner self‐regulation: Enhancing achievement in the first year using learning technologies. Assess. Eval. High. Ed. 2009, 34, 334–352, DOI: 10.1080/02602930802255139. 28. Niroj, S.; Srisawasdi, N. A blended Learning Environment in Chemistry for Promoting Conceptual Comprehension: A Journey to Target Students’ Misconceptions. Paper presented at 22nd International Conference on Computers in Education, Nara, Japan, 2014. 29. Wiggins, G. P.; McTighe, J. Understanding by design, 2nd ed.; Association for Supervision and Curriculum Development: Alexandra, VA, 2005. 30. Azevedo, R.; Hadwin, A. F. Scaffolding self-regulated learning and metacognition – implications for the design of computer-based scaffolds. Instr. Sci. 2005, 33, 367–379, DOI: 10.1007/s11251-005-1272-9. 31. Means, B.; Toyama, Y.; Murphy, R.; Bakia, M.; Jones, K. Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies; U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, 2009. 32. Pea, R. D. The social and technological dimensions of scaffolding and related theoretical concepts for learning, education, and human activity. J. Learn. Sci. 2004, 13, 423–451, DOI: 10.1207/s15327809jls1303_6. 33. Chiu, M.-H.; Wu, H.-K. The roles of multimedia in the teaching and learning of the triplet relationship in chemistry. In Multiple representations in chemical education; Gilbert, J. K., Treagust, D. F., Eds.; Springer: Dordrecht, Netherlands, 2009, pp 251-283. 34. Nicol, D. J.; MacFarlane-Dick, D. Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Stud. High. Educ. 2006, 31, 199–218, DOI: 10.1080/03075070600572090. 35. Mayer, R. E. Cognitive theory and the design of multimedia instruction: An example of the two-way street between cognition and instruction. New Directions Teach. Learn. 2002, 55–71, DOI: 10.1002/tl.47. 36. Mayer, R. E.; Moreno, R. Nine ways to reduce cognitive load in multimedia learning. Ed. Psych. 2003, 38, 43–52, DOI: 10.1207/S15326985EP3801_6. 37. Belland, B. R. Scaffolding: Definition, current debates, and future directions. In Handbook of Research on Educational Communications and Technology, 4th ed.; Spector, M. J., Merrill, M. D., Elen, J., Bishop, M. J., Eds.; Springer: New York, 2014; pp 505−518. 38. Puntambekar, S.; Kolodner, J. L. Toward implementing distributed scaffolding: Helping students learn science from design. J. Res. Sc. Teach. 2005, 42, 185–217, DOI: 10.1002/tea.20048. 39. Tabak, I. Synergy: A complement to emerging patterns of distributed scaffolding. J. Learn. Sc. 2004, 13, 305–335, DOI: 10.1207/ s15327809jls1303_3. 19 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

40. Quintana, C.; Zhang, M.; Krajcik, J. A framework for supporting metacognitive aspects of online inquiry through software-based scaffolding. Ed. Psych. 2005, 40, 235–244, DOI: 10.1207/s15326985ep4004_5. 41. Quintana, C.; Reiser, B. J.; Davis, E. A.; Krajcik, J.; Fretz, E.; Duncan, R. G.; Kyza, E.; Edelson, D.; Soloway, E. A scaffolding design framework for software to support science inquiry. J. Learn. Sci. 2004, 13, 337–386, DOI: 10.1207/s15327809jls1303_4. 42. McRae, C.; Karuso, P.; Liu, F. Chemvoyage: A web-based, simulated learning environment with scaffolding and linking visualization to conceptualization. J. Chem. Educ. 2012, 89, 878–883, DOI: 0.1021/ ed200533u. 43. Al-Balushi, S. M.; Al-Hajri, S. H. Associating animations with concrete models to enhance students’ comprehension of different visual representations in organic chemistry. Chem. Educ. Res. Pract. 2014, 15, 47–58, DOI: 10.1039/c3rp00074e. 44. Johnstone, A. H. Why is science difficult to learn? Things are seldom what they seem. J. Comp. Assist. Learn. 1991, 7, 75–83, DOI: 10.1111/j.13652729.1991.tb00230.x. 45. Tasker, R. Visualising the molecular world for a deep understanding of chemistry. Teach. Sci. 2014, 60, 16–27. 46. Tasker, R. Vischem. http://www.vischem.com.au (accessed August 31, 2016). 47. Tasker, R.; Dalton, R. Research into practice: Visualisation of the molecular world using animations. Chem. Educ. Res. Pract. 2006, 7, 141–159, DOI: 10.1039/B5RP90020D. 48. Bhattacherjee, A.; Perols, J.; Sanford, C. Information technology continuance: A theoretic extension and empirical test. J. Comput. Inform. Syst. 2008, 49, 17–26, DOI: 10.1080/08874417.2008.11645302. 49. Lawrie, G.; Wright, A.; Schultz, M.; Dargaville, T.; O’Brien, G.; Bedford, S.; Williams, M.; Tasker, R.; Dickson, H.; Thompson, C. Using formative feedback to identify and support first year chemistry students with missing or misconceptions. A practice report. Int. J. First Year High. Ed. 2013, 4, 111−116. DOI: 10.5204/intjfyhe.v4i2.179. Creative Commons Attribution Licence (CC-BY) link: https://creativecommons.org/licenses/by/3.0/at/ deed.en_GB. 50. Schultz, M.; Lawrie, G. A.; Bailey, C. H.; Bedford, S. B.; Dargaville, T. R.; O’Brien, G.; Tasker, R.; Thompson, C. D.; Williams, M.; Wright, A. H. Evaluation of diagnostic tools that tertiary teachers can apply to profile their students’ conceptions. Int. J. Sci. Ed.submitted. 51. Briggs, D. C.; Alonzo, A. C. The Psychometric Modeling of Ordered Multiple-Choice Item Responses for Diagnostic Assessment with a Learning Progression. Paper presented at Learning Progressions in Science, Iowa City, IA, 2009. 52. Hadenfeldt, J. C.; Bernholt, S.; Liu, X.; Neumann, K.; Parchmann, I. Using ordered multiple-choice items to assess students’ understanding of the structure and composition of matter. J. Chem. Educ. 2013, 90, 1602–1608, DOI: 10.1021/ed3006192. 20 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch001

53. Ainsworth, S.; Newton, L. Teaching and researching visual representations: Shared vision or divided worlds? In Science Teachers’ Use of Visual Representations; Eilam, B., Gilbert, J. K., Eds.; Springer: Cham, Switzerland, 2014; pp 29−49. 54. Mousavi, S. Y.; Low, R.; Sweller, J. Reducing cognitive load by mixing auditory and visual presentation modes. J. Ed. Psych. 1995, 87, 319–334, DOI: 10.1037/0022-0663.87.2.319. 55. McIntyre, R.; Wegener, M.; McGrath, D. Five Minute Physics. http://teaching.smp.uq.edu.au/fiveminutephysics/ (accessed June 17, 2016). 56. Wegener, M.; McIntyre, T.; McGrath, D.; Talbot, C. Concise, Interactive e-Learning Modules for Student Lecture Preparation. Paper presented at The Australian Conference on Science and Mathematics Education, Canberra, Australia, 2013. 57. Phillips, R.; McNaught, C.; Kennedy, G. Evaluating e-learning: Guiding research and Practice; Routledge: New York, 2012. 58. Tasker, R. ConfChem conference on interactive visualizations for chemistry teaching and Learning: Research into practice - Visualizing the molecular world for a deep understanding of chemistry. J. Chem. Educ. 2016, 93, 1152–3, DOI: 10.1021/acs.jchemed.5b00824.

21 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Chapter 2

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

Improving Academic Reading Habits in Chemistry through Flipping with an Open Education Digital Textbook Brett M. McCollum* Department of Chemistry and Physics, Mount Royal University, Calgary, Alberta, Canada T3E 6K6 *E-mail: [email protected]

Improving first-year general chemistry student reading habits was the motivation for a systemic redesign of instructor-student-content interactions. A ChemWiki open-education hyper-textbook was designed for course readings, and became a focal point for student learning through a flipped classroom, weekly academic reading circles, and an online learning system. The use of pre-lecture videos was avoided in favor of reading assignments. Additional technology, including mobile apps, were used to further enhance engagement. To strengthen the peer relationships emerging from the team-based learning, formative assessments were redesigned. Dramatically improved reading habits resulted from this set of complementary interventions.

© 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

Introduction For several years I taught following the approach of Collard, Girardot and Deutsch, providing “skeleton” notes, which students downloaded from the course learning management system (LMS) prior to the lecture (1). Students were generally happy with the approach, watching as I attempted to deliver a dynamic lecture interjected with clicker questions and brief team-based learning activities (2). My department chair and dean have been pleased with my student evaluations of instruction. Based on my annual reviews, I was exceeding expectations at a teaching-focused undergraduate university. However, it was always apparent to me that the majority of my students were completely reliant on me for their learning, devoting only the minimal amount of time outside of class to their chemistry studies. In particular, I was troubled by the inability of my students to properly describe chemical phenomena, the problem apparently stemming from their unwillingness to read the assigned textbook and strengthen their familiarity with chemical jargon. Despite our best intentions, which included assigning reading each class and providing associated learning objectives, my peers and I found that our students expected that any testable course content be covered in lecture to the same complexity of examination questions. The expensive course textbook was treated as a source of additional practice problems or a reference in emergency purposes only, not as a core learning resource. A review of the literature reveals that we were not unique in our experience (3–6). For example, Yonker & Cummins-Sebree report that “a majority of traditional students are ill-equipped or unwilling to read the essential amount of text” (p 169), with 42% of participants reading less than one-quarter of the assigned material (7). The Carnegie credit hour states that, for every one hour of classroom instruction, students should be spending, at a minimum, two hours outside of the course, either preparing for upcoming lectures or completing course assignments (8). However, university students are spending less time studying than attending class, and 3 times fewer hours studying than on socializing or other forms of entertainment (9, 10). To better understand the scope of the problem, I surveyed my students about their reading habits in first-semester general chemistry. While the quantitative data confirmed my concerns (less than 3% of the students were completing the assigned readings ahead of class; another 11% after class), the qualitative comments were distressing in their frankness. Student C1: “Opened it once but it seemed complicated.” Student C2: “Reading is a waste of time.” Unfortunately, these comments were not isolated cases, but the rule. Many of the student responses identified difficulties with understanding the academic text, or an opinion that the text was TLDR – Too Long; Didn’t Read. The potentially negative impact on student learning of indiscriminate use of PowerPoint-style presentations has been explored (11–14). Students have little incentive to read 24 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

course materials which will be outlined by the professor later in class (15). It became apparent to me that my seemingly successful strategies were enabling the poor academic habits of my students. While they were happy with my efforts, and I was performing to the department’s expectations, my students were missing out on developing a vital skill during their introductory university experience: learning how to critically read an academic text. In my opinion, literacy remains a core component of higher education. Evidence suggests students can be successful when provided appropriate scaffolding (16). In this chapter I will share my experience using a personalized open-education digital textbook, with appropriate scaffolds and assessments, and how I have created a learning environment where students are engaging with the course text in a meaningful way.

Context Building on its 100 year history as a two-year college, Mount Royal University (MRU) became a four-year degree-granting undergraduate-only institution in 2009. Most classes on campus have less than 30 students, the major exception being those in the Faculty of Science and Technology where practical laboratory experiences are common and lecture sections are often 40 – 75 students. In Fall 2015, I taught three sections of General Chemistry I (CHEM 1201), each with 40 – 58 students. Two sections served as my control, using a commercial textbook and active lectures. The third section was my experimental section, using an open-education online digital textbook and an inverted or flipped instructional approach (17, 18) with weekly academic reading circles (19, 20). A fourth section was taught in Winter 2016 using the experimental approach. All sections were assigned online learning assignments through OWLv2 from Nelson/Cengage. Classes were run at approximately the same time of day for all sections (between 12:30pm and 3:30pm) and met two days each week for 80 minutes. In this course, tutorials and laboratories are normally de-linked from lectures. This resulted in a mixing of students from all lecture sections, including those not taught by the author, in these learning environments. Thus, the learning innovations only applied to the lecture component of the course. The gender distribution of all sections was similar to the university student population (36% male, 64% female). The age distribution, with the majority of students 18 – 20 years old, was comparable for the traditional and flipped general chemistry sections. All student quotes within this chapter are attributed to study participants from either the control (C) or experimental (E) sections.

Adopting an Open-Education Textbook The first challenge I faced in getting my students to meaningfully engage with their course textbook was to get a copy of the common text into their hands. Modern texts are typically printed in full color on glossy paper, and come with a multitude of digital supporting materials for both students and instructors. 25 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

However, these additional resources have come at a cost, and affordability of commercial textbooks has been identified as a barrier for lower income students (21, 22). Open Educational Resources (OERs) have been identified as a potential solution to accessibility in light of rising textbook costs (21, 23). The STEMWiki Hyperlibrary Project is based at the University of California Davis (24). Similar to other wikis, the content is covered by a creative commons license. The key difference is in terms of quality control: only those with authoring accounts (faculty at various universities and the ChemWiki student volunteers at UC Davis) have authority to edit the wiki. ChemWiki, the core module of the STEMWiki, has been successfully adopted by faculty as a replacement for commercial textbooks (25–28). With the help of STEMWiki Director, Delmar Larsen, I began curating existing content and creating new material when necessary to build a textbook for my course (29). Adopters also have the option to use an existing ChemWiki textbook. The ability to duplicate pages in use by other faculty provided me with the capability to personalize my text without imposing my approach to content on others. Personalization of the text, such that it is tailored to flow effectively with other elements of the course and convey the instructor’s voice, has been described as a benefit of OERs (30). In addition, creating my own ChemWiki textbook allowed me to embed my course learning outcomes directly within the textbook. Whereas my ChemWiki text is a digital textbook, I was curious about the consequence of digital reading as opposed to reading print text. A number of faculty at my institution remain skeptical of digital texts, reflecting a widely held opinion (31). These opinions appear to be based more in tradition than evidence. Although the manner of presentation may influence physical behaviors while reading (32), such as following the text with a finger or cursor, or silently mouthing the words, there is evidence that comprehension is unaffected (33). However, there are opposing views, depending on the digital technology used by the reader (34). While the duration of fixations can vary (longer for young adults and shorter for older readers), regressions are unchanged (35, 36). Most importantly, it has been demonstrated that cognitive learning from digital textbooks is comparable to their print counterparts (37). While it appears that digital textbooks do no harm, there are significant potential benefits. In addition to zero cost for the textbook, my students have appreciated the ability to access the text through the internet-connected devices they already own. Mobility has also been identified as an advantage, with students reading the text on the bus or at the shopping mall. For instructors looking to transform reading assignments into an ideal mLearning (mobile learning) experience, the transition to digital texts is a requirement (38). Adopting ChemWiki as my textbook resolved the issue of getting a common text into my students’ hands. However, there is no guarantee that they will read the textbook. As a friend, colleague and OER advocate recently mused, “what is the pedagogical benefit of a free textbook that no one reads as compared to a commercial textbook that no one reads?” (39). Appropriate scaffolds are required to instigate and support student engagement with academic texts. In my 26 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

experience, these scaffolds include flipped or inverted teaching and academic reading circles.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

Flipped Learning The spectrum of instructional methods that can be associated with the flipped learning movement is fairly broad (40, 41). This includes peer-led team learning (PLTL) (42), peer instruction (43), inverted classrooms (17), just-in-time teaching (JiTT) (44), learn before lecture (45), teaching naked (46), and flipped classroom (18). Flipped methods have been embraced by numerous chemistry educators, with a number of recent publications devoted to the topic (28, 47–58). The impact of flipped instructional methods on student achievement has been variable (50, 52). For example, Bradley, Ulrich and Jones offered sophomore organic chemistry students the choice of flipped or traditional when they enrolled in the second semester of the course (59). They observed that the flipped approach did not affect the grade distribution in either positive or negative directions. In contrast, Yestrebsky’s comparative study in first-year general chemistry illustrated a flipped method that successfully supported B- and C-level students in performing at Aand B-level standards, respectively (57). However, Yestrebsky noted that the approach did not reduce the number of D and F grades. One possible interpretation of the mixed influence is that the success of flipped approaches depends on the particular instructional design choices and the level of prior knowledge of the students. Based on the expertise reversal effect, techniques that engage high-prior knowledge learners are often too challenging for low-prior knowledge learners, while instructional methods that are designed for the latter group typically disinterest the former (60, 61). By extension, this suggests that specific choices in the approach of Bradley, Ulrich, Jones and Jones with the student population at Princeton (59), a highly selective university, will not have the same effect in lower prior-knowledge populations. Hence, confusion around the benefits of flipping persists. Reports on the efficacy of active-learning methods in chemistry, such as flipped instruction, should include information on the level of prior knowledge in the sample population. In practice, this may be difficult unless an appropriate standard is identified for all researchers, such as the Toledo placement examination from the American Chemical Society Division of Chemical Education (62). However, such examinations are not best aligned with incoming student knowledge at all levels or in all jurisdictions, and some institutions may not be comfortable publishing the scores of incoming students. Ryan and Reid provide a case of traditional and flipped second-term general chemistry classrooms that, while yielding statistically identical exam performance across the sections, resulted in decreased DFW rates (63). Thus, flipped instructional methods can be designed to support the learning of the lower quartile. However, the benefits observed by Yestrebsky for the middle quartiles were not detected by Ryan (57, 63). 27 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Many flipped classrooms have used online videos (voiced over PowerPoint) to support their students’ learning. Considering that my goal is to create a learning environment where students are engaging with the course text in a meaningful way, instead of extensive reliance on pre-lecture videos, a deeper reflection on flipped instructional methods is appropriate.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

Text-Centric Flipping Flipped instruction, as described by Bergmann and Sams involves having students watch pre-recorded lectures before class to prepare for the active learning environment (18). While not all flipped or inverted approaches use online videos,59 it has become the standard method (49–58). Herreid and Schiller report that the majority of teachers prefer using online videos over reading materials for student pre-class preparation, and students prefer videos too (64). However, this approach contains significant pitfalls. Poh, Swenson & Picard demonstrated that watching videos and attending lectures are among the least attention-demanding tasks we can ask of learners (65). In contrast, they observed elevated electrodermal activity – which they associate with higher attention-demanding tasks – when participants were studying, doing homework, or writing an exam. If pre-recorded online lecture videos are simply the combination of two low attention-demanding tasks (65), then indiscriminant use of pre-lecture videos may prove to be as harmful to student learning as indiscriminant use of PowerPoint (11–14). Considering the current popularity of the method, additional research is needed on this important issue. When faculty predigest curriculum content for students in the form of PowerPoint lectures or online videos, are they simply enabling poor study habits (9, 10)? Discussion on the use of pre-class videos for flipped learning at the Symposium on Scholarship of Teaching and Learning exposed conflicting opinions on the value of academic texts in the era of online video streaming (66). A question that emerged was: can university-level students successfully read a university-level text? This concern was echoed in the unrelated community of ConfChem (26). That this question is being asked by faculty across many disciplines is alarming! It is the responsibility of university educators and administrators to facilitate an environment where students are appropriately motivated and supported to develop an academic level of reading proficiency. Recall that students can critically read academic texts when provided appropriate scaffolding (16). While I do not consider all pre-lecture videos to be a problem, those that are not supported by active learning tools undermine the importance of text. Anecdotally, in my experience these enablers can result in transfer of negative habits between courses, increasing resistance in learners against active-learning approaches. Upon reflection, in order to achieve my goal of creating a learning environment where students are engaging with the course text in a meaningful way, I began taking the following actions: 28 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

• • •

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

•

Adopting an online OER textbook (ChemWiki) to reduce barriers Promoting the value of the text by making it central to in-class activities and assigning readings after each class Inverting / flipping the classroom to reduce debilitating student reliance on passive learning experiences Facilitating additional scaffolding that supports learning of the lower quartile while also serving the rest of the class

Despite observations that it is the active-learning that yields the benefits of flipped instructional techniques (67), I recognized that additional scaffolding was required to properly support my students as they engaged with the text. Structuring an active-learning classroom was not sufficient. After all, many of my students were reporting that they were not capable of, or not willing to engage in, academic reading. Peer discussion on the assigned readings through organized academic reading circles was the option I chose.

Academic Reading Circles Academic Reading Circles (ARCs) – also known as reading circles, peer discussion groups, or literature circles – have been used to stimulate discussion and help learners construct meaning from a common text (19, 20). While ARCs have primarily been employed in K-12 English classrooms, there have been several adaptations for second language (L2) courses (20, 68–70). Much of the formalization of ARCs is based on the work of Daniels (19). The key elements of a successful literature circle as described by Daniels (p 18), with updates from Shelton-Strong (20), are: • • • • • • • •

variability in temporary groups student-chosen books (if appropriate) discussions occurring regularly in an open and natural manner students choosing discussion topics use of notes or drawings by the students for guidance the teacher as a facilitator student-focused evaluations a spirit of playfulness and fun

In L2 instruction, the structure of a formal ARC consists of the mutual text, assigned student roles, and the group discussion (70). Adjustments were needed to adopt the ARC method for a chemistry course, including a reduction in the length of time for the group discussion and adaptation of the member roles. Seburn advocates for 60-90 minute uninterrupted group work, followed by further class-wide discussion led by the instructor (p 20) (70). Whereas my classes ran for 80 minutes twice a week, I needed to reduce the ARC open-discussion to a much shorter time frame. Through trial and error, I found that within 1015 minutes most groups were able to discuss the reading in terms of topics they understood, were challenged by, or did not understand. Next, I brought the class 29 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

together to resolve outstanding questions related to the reading assignment, and then the ARC groups engaged in content-driven problem solving. The ARC member roles I used were Leader, Contextualizer, Visualizer, Connector and Highlighter. Based on the course curriculum and learning objectives, duties of the roles were revised. For example, placing the text within social contexts was less important, but emphases on chemical and mathematical representations were needed. A description of each revised role follows.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

Leader: Facilitates discussion, leading toward agreement on the key points of the assigned reading. Duties: summarizes the main ideas at the start and end of discussion, and guides the group discussion through the reading, inviting members to share what they have learned in their role Poses questions on: the overarching themes and ideas concepts emerging from the reading how group members would summarize particular key ideas from the reading Contextualizer: Explains why the author refers to other topics, concepts, people, or dates for support. Duties: learns about the context surrounding the reading, and determines which ideas are foci (useful) and which are tangential (insignificant) Shares information on: specific people, places, events, or ideas that the author mentions as support identifies useful and insignificant contextual references Visualizer: Uses visuals to facilitate language used in descriptions and explains visuals that accompany the text, including graphical representations of data. Duties: explains visuals provided in the text, linking them to associated sentences/ paragraphs determines if provided visuals support comprehension or are primarily decorative / expand beyond the key ideas 30 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Create visual representations of concepts in the text, such as mind maps, word clouds, graphs, charts, or disciplinary (chemistry) representations Shares information on: the relationship between visual representations and the text Connector:

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

Creates meaningful connections between concepts in the assigned reading, equations, and related/familiar situations. Duties: identify connections between the assigned reading and past concepts, personal experiences, or current news/events Shares information on: the relationship between equations and the text concepts in the assigned reading and past course concepts, or material from other courses concepts in the assigned reading and current news/events or personal experiences Highlighter: Facilitates lexical comprehension, raises awareness of topical vocabulary, and explains mathematical representations. Duties: prepare a list of key topical vocabulary, defining each term, identifying its placement within the text, and linking it to other ideas define the variables within equations and illustrate how to perform the necessary calculations Shares information on: key topical vocabulary equations within the assigned reading To help learners prepare for the ARC discussions, students were presented with the SQ3R (Survey, Question, Read, Recite, Review) process for reading.71 Users of the SQ3R process proceed through the following steps: • • • • •

surveying the text to develop an overview perspective generate a question based on each section heading to focus their reading actively read each passage to identify relevant information and answer their questions reciting the answers to the questions in their own words review the main concepts to facilitate long-term memory development 31

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

Students were specifically instructed to perform their recitations in written form, not mentally. This adheres to the original instructions for SQ3R and aligns with what is known about writing to improve reading skills (72, 73). I have observed that ARCs motivate students to complete their reading assignments, particularly as they develop relationships with their peers. General chemistry students have identified the ARC groups as the first time they talked to peers in any of their courses. This was true for both my Fall and Winter semester flipped classes. Students admitted that their isolation is self-imposed through the use of headphones and music to avoid talking to others, a documented behavior (74). This self-isolation behavior is likely also a characteristic of my institution being a commuter campus (75). ARCs, along with the other active-learning aspects of my inverted classroom, function to combat student isolation as recommended by Funk (76): “Commuter students may not have the luxury or desire to spend more time on campus except to attend classes. In order to combat feelings of isolation and disconnectedness for commuter students, college administrators [and faculty] should strive to ensure meaningful learning, such as active learning experiences incorporated in the classroom during class time” (p 99). Peer relationships, developed through ARCs, serve another important purpose: learners become comfortable exposing weaknesses in their comprehension. Similar to Grisham and Wolsey (77), I observed that discussions in some ARCs were fairly superficial without instructor intervention. These learners appeared hesitant to engage in meaningful discussion of material they had not mastered, even though that was the purpose of the ARC. I adjusted my instructions for ARC preparation to have students also record all questions they had about the material. Questions they had not resolved in their own study could then be posed to their ARC and the group would attempt to construct a meaningful answer, thus supporting each member’s learning. The key element was that they were expected to share their questions, opening up to their peers about their struggles with the course content. This small change in practice helped less engaged ARCs function better, and quieter ARC members to more fully participate. At this time, I am unsure if the full impact of the ARCs is obvious to students in my inverted classes. However, by teaching both control and experimental sections, from my perspective I see significantly different attitudes toward the textbook and learner responsibilities. A comparison of attitudes towards reading between my regular and inverted sections is illustrative. Consider student C3 from a Fall 2015 lecture section taught using my traditional approach. Student C3 “Doing questions is more helpful than reading.” This student has not participated in an ARC. On the survey he/she reported only using their textbook for practice problems before an exam. For the control section students, among those that opened their textbook at least once, this was a common response. In contrast, consider students E1 – E3 who were in the Fall 2015 inverted class. 32 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Student E1 “Chemwiki is my primary study resource. I transfer what I am reading into my notebook making sure to properly paraphrase.” Student E2 “I’ve tried more lately to take my own notes when reading and have questions ready for class.”

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

Student E3 “The art of reading to prepare for class helps [me] to come to class ready to ask questions and know what we are going to do.” Notice the positive study habits identified by student E1, and the emphasis on generating questions in preparation for class from students E2 and E3. In a text-centric inverted or flipped classroom, ARCs successfully focus the student preparation on the textbook. As learners build a sense of community within their ARC they are willing to discuss with peers the challenges they face when critically engaging with an academic text.

The Role of Technology in a Text-Centric Classroom Using a web-based OER for my course textbook, internet connected technologies (ICTs) were important elements of my classroom. ChemWiki runs well in any web-browser, permitting a BYOD (bring your own device) learning environment. Smartphones are ubiquitous on our campus, and many students also own a laptop and/or tablet. Thus, all students in the inverted classes were able to access the textbook on ChemWiki, and other learning materials on the course LMS (Blackboard) through their personal devices. Laptops and iPads were the most commonly used technologies, but some students did access content through their smartphones. Most ARCs would load ChemWiki on a member’s laptop, navigating the reading assignment as part of their discussion. Conversations often focused on figures, equations, or chemical representations, and their relationship to the written text. Interestingly, more than half the class maintained their reading and class notes in a paper notebook. Those that used their electronic devices for notetaking had a touch-screen tablet or laptop that facilitated sketching the chemical representations common in an introductory course on chemical bonding models. When selecting a textbook, one consideration for faculty is level of content and the aforementioned expertise reversal effect (60, 61). ChemWiki is no different, as illustrated by the following contrasting comments from two learners in my inverted classes. Student E4 “Personally, reading helps [me] prepare for class. Sometimes I find that pre-lecture material for [name of other course] goes into too much detail. However, ChemWiki is very concise and does not burden the reader with superfluous details. When Brett highlights key terms in preparation to a reading it helps me, A LOT, since I need a lot of structure to effectively organize my notes and study key points in a chapter.” 33 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

Student E5 “My only wish [is] that there be more to read and more detailed explanations as well.” These two students are requesting two different textbooks, the first wanting a concise textbook and the second desiring a more in-depth textbook. As a web-based OER, ChemWiki has the potential to partially resolve the expertise reversal effect in a way not possible with static texts. Computer code has been identified and successfully implemented in my ChemWiki hypertext that allows ‘accordion’ content, additional information that can expand out on demand as shown in Figures 1 and 2 (29). This feature permits learners to attempt an exercise without accidentally seeing the solution, and avoids the use of excessive hyperlinks. While hyperlinks support non-linear learning, allowing students to make connections to other concepts or perspectives (78), hyperlinks place a higher cognitive load on users and are not beneficial to low- or high-prior knowledge learners (79).

Figure 1. ChemWiki ‘accordion’ content in its initial collapsed state. In my first use of ChemWiki I used this ‘accordion’ content feature only for solutions to exercises. Based on the survey responses from students in my inverted classes (e.g. Students E4 & E5), I plan on selectively expanding the hypertext through the use of expandable descriptions that are optional reading for students who want additional explanation or examples. All students and faculty at MRU have institutional Google accounts, including access to Google Drive, Google Docs, and more. Shared Google Docs have been used for ARCs to compile questions in preparation for class. This approach can support JiTT (44), allowing the instructor to select class activities based on student questions or misconceptions. Initially, I created the documents and shared them with each ARC, but later moved toward having the group make their own shared document, which they would share with me. Perhaps because no marks were assigned to the shared Google docs, they were not well used. In future semesters I am considering having students submit their reading records (SQ3R and list of content questions) through shared Google docs so that their ARC can provide additional support out of the classroom. This approach would be similar to asynchronous forums, which have been used for ARCs (77), but also allow synchronous editing of the Google doc. Additional technology used during class time included numerous apps, such as the Explain Everything tablet app, coupled with Reflector to broadcast student work to the class through the classroom video projection unit. The Molecules app 34 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

for iPad can be combined with physical models for learning VSEPR Theory (80, 81), and was highly engaging as a team-based learning activity.

Figure 2. ChemWiki ‘accordion’ content in its user triggered expanded state.

Integrating mobile social learning (mLearning) into the ARC or flipped classroom has the potential for even greater impact (38). For example, students with iPads were able to use a shared virtual whiteboard through the Talkboard app. This was used in class during Fall 2015, but the classroom did not have sufficient WiFi bandwidth. Students reported using the app, including the microphone feature for voicechat, to run virtual study sessions from home with the shared whiteboard. Between the ARC discussions and group problem solving, I would bring the class together to resolve outstanding questions. Displaying the ChemWiki textbook on the projector screen permitted me to demonstrate the centrality of the text in the course, which was particularly important in the first few classes. After the problem solving activity, and before the end of class, I gave a brief ‘mini-lecture’ on the content they would read before the next class. This typically focused on identifying terminology they would need to define, listing the topics they would explore, and providing context for more challenging concepts. This approach allowed me to shift between facilitator and instructor as appropriate for a constructivist approach to learning (17, 82–84). 35 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

The online learning system, OWLv2 from Nelson/Cengage, was used to promote reading and support self-assessment of content comprehension. Although students were encouraged to complete the ChemWiki reading first and then attempt the OWLv2 assignments, many reported using both resources side-by-side. Further support for the inverted classroom was revealed through the OWLv2 grades. The inverted students were required to complete their online learning assignments prior to associated in-class problem solving session, while the students in the control sections completed their assignments after the content had been taught in lecture. Students in the inverted sections were more likely to take advantage of the multiple attempts provided on questions to master the content, resulting in a higher class average (about 10% higher) for the inverted sections relative to the control sections.

Strengthening Social Collaboration through Assessments ICTs play a central role in my inverted classroom, including the ChemWiki hypertext, the other learning materials delivered through the course LMS, the online learning system, and more. However, when it came to most in-class assessments I took a low-tech path. To maintain comparability with other sections (both my control sections, and additional sections taught by other instructors) I used the same in-term and final examinations for summative assessment. In contrast, the weekly individual or group quizzes that I used as formative assessments were replaced in the inverted sections with group unit quizzes based on the scratch card quizzes of Michaelsen and Sweet (85). With groups of approximately five students in a class of up to 60 students, I was able to design the group quizzes to be open-response rather than multiple-choice. An example question is shown in Figure 3. Teams were required to generate an ideal solution to earn a grade on a problem. A successful response on their first attempt was granted 4 points. This was then reduced to 2 points, and 1 point, on successive attempts, with a limit of three attempts per question. Next to ChemWiki, these group multi-attempt quizzes were the most popular element of the experimental sections. Students reported that in addition to the team-based problem solving and ARCs, the group quizzes further strengthen their sense of community, similar to the observations of Sweet and Pelton-Sweet (86). After the first group quiz, it was observed that that students began attending the Chemistry Study Center for self-initiated group study sessions.

36 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

Figure 3. Using a team-based learning technique, groups of approximately five students worked together on open-response unit quizzes. Each quiz was 8 questions long, and groups were provided roughly 30 minutes to complete the quiz. Grading was done throughout the quiz time, with more points awarded for fewer attempts.

37 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

Observed Changes in Behavior Group multi-attempt quizzes, online learning systems, academic reading circles, and inverting my classroom with a web-based open education hypertext; all of these changes were implemented to work together as a systemic intervention. The goal was simple: fulfill my responsibility as a professor and inspire my students to critically engage with academic texts. While some measures of my approach’s success (or lack thereof) are still being assessed, a few salient evidences are available at this time. First, consider the anonymous self-reported reading habits shown in Figure 4. Despite regularly discussing the importance of textbook reading in preparation for class with my students in the two control sections, the change in the fraction of students that completed their pre-lecture reading was insignificant (Fall 2014 vs. Fall 2015 Control). In contrast, the behavior of students in the inverted section was astoundingly dissimilar with 95% of students reporting that they read in preparation for class and the remaining 5% sometimes completing the reading after class.

Figure 4. In each of the semesters shown, traditional active learning methods (Fall 2014 three sections nTotal = 112, and Fall 2015 Control two sections nTotal = 109) have resulted in less than five percent of students completing recommended pre-class readings. In the inverted section (Fall 2015 Experimental one section n = 40), 95% of learners self-reported regularly completing the pre-class readings. Reporting was anonymous. All sections were taught by the author. This dramatic improvement in reading habits was verified through Google Analytics, looking at the readership of my ChemWiki hyper-textbook. When the data are localized to Calgary, the hyper-textbook has demonstrable upticks in number of page views on the mornings of class (the class met at 2:00-3:20pm on Monday and Friday) and a total reading time by Calgary users during the Fall 2015 semester that can be estimated as 28 hours per learner. Yet, I recognize 38 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

that by not restricting the hypertext to course registrants, there may have been other users within the city, such as students in other sections of CHEM 1201 at MRU, students in other courses at the university looking for refresher material, or students at the other local university or colleges. It is worth noting that the hyper-textbook did have significant external reach. As the sixth most accessed hypertext on ChemWiki, over 80% of the 5300 monthly unique page views were by users outside of Calgary. An interesting measure of this intervention’s success relates to the dependence of learners on the course instructor. In my normal student evaluations of instruction, optional student comments almost universally address my enthusiasm for chemistry and my teaching methods. In the Fall 2015 experimental section, my quantitative score for General Evaluation of Instructor decreased slightly (0.15 on a 5 point scale relative to a five-year average, but within error bars) and less than 40% of the student comments were about what I did in class. The other 60% of student comments were about what they did in the class. Topics covered peer relationships, ARCs, team-based learning and assessment, the ChemWiki hyper-textbook, online learning assignments, and instructional time use. This reveals important self-reflections by my students on how the instructional elements discussed in this chapter facilitated their learning. The comments are a reflection of stronger peer relationships and a reduced reliance on me. This is similar to the observations of Yestrebsky (57). A final reflection relates to my own experience in implementing this collection of complementary interventions. The practical and emotional demands of this systemic innovation were significant (87). Besides the creation of the ChemWiki hyper-textbook, additional time investment was required to create or adapt learning materials for the inverted classroom experience and to structure the associated online learning assignments. Regular preparation of lesson plans was key to maintaining my organization. Facilitation of the inverted classroom also required additional focus and energy, with all the extra stimulation (for both students and myself) in an active-learning environment. As a faculty member at an undergraduate teaching-focused university, balancing the demands of teaching and research is always a challenge. While the initial setup of this set of interventions was demanding, the reusability of course materials in Winter 2016 has proven worth the investment. Changing my practice from active lecturer to learning facilitator has been a fruitful experience.

Conclusions A systemic innovation, constructed from a set of complementary and proven interventions, was found to dramatically improve student reading habits. Learners regularly engaged with a faculty-personalized online open education textbook (a ChemWiki hyper-textbook), both for personal study and in the classroom. Academic reading circles and pre-class reading preparations were used to further support critical reading, although changes were required to adopt ARC methods for a HiEd chemistry course. 39 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

By implementing this set of complementary interventions, I have successfully created a learning environment where students are engaging with the course text in a meaningful way. Additionally, facilitation of team-based learning activities, group multi-attempt quizzes and academic reading circles in an inverted / flipped classroom, led to stronger relationships between peers. In the near future, I aim to answer the remaining question, of whether these interventions support learning of the lower quartile while also serving the rest of the class. Additional data is required to make a conclusion regarding grades and whether these activities increase student inclination to engage with academic texts in other courses. However, other indicators of academic learning, such as engagement with the textbook and online learning system, and reports of positive peer collaboration, point in the right direction.

References 1.

2.

3.

4. 5. 6. 7. 8.

9. 10. 11. 12. 13. 14.

Collard, D. M.; Girardot, S. P.; Deutsch, H. M. From the textbook to the lecture: improving prelecture preparation in organic chemistry. J. Chem. Educ. 2002, 79, 520–523. Michaelsen, L. K. Team learning: A comprehensive approach for harnessing the power of small groups in higher education. To Improve the Academy 1992, 11, 107–122. Simpson, M. L.; Nist, S. L. In Comprehension Instruction: Research-Based Best Practices; Block, C., Pressley, M., Eds.; Guilford: New York, 2002; pp 365–379. Lord, T. Darn it, Professor. Just tell us what we need to know to pass your course. J. Coll. Sci. Teach. 2008, 37, 71–73. McCormick, A. C. It’s about time: What to make of reported declines in how much college students study. Liberal Education 2011, 97, 30–39. Culver, T. F. Increasing Reading Compliance and Metacognitive Strategies in Border Students. J. Coll. Read. Learn. 2016, 46, 42–61. Yonker, J. E.; Cummins-Sebree, S. To Read Or Not To Read: How Student Characteristics Relate To Textbook Reading. AURCO J. 2009, 15163–15172. Silva, E.; White, T.; Toch, T. The Carnegie Unit: A Century-Old Standard in a Changing Education Landscape; Carnegie Foundation for the Advancement of Teaching: Stanford, CA, 2015. Brint, S.; Cantwell, A. M. Teach. Coll. Rec. 2010, 112, 2441–2470. Arum, R.; Roska, J. Academically Adrift: Limited Learning on College Campuses; University of Chicago Press:Chicago, IL, 2011. Adams, C. PowerPoint, habits of mind, and classroom culture. J. Curriculum Stud. 2006, 38, 389–411. Adams, C. On the “informed use” of PowerPoint: Rejoining Vallance and Towndrow. J. Curriculum Stud. 2007, 39, 229–233. Clark, J. PowerPoint and pedagogy: Maintaining student interest in university lectures. Coll. Teach. 2008, 56, 39–45. Mitchell, T.; Hutchinson, S. In Southern Regional Council on Educational Administration 2012 Yearbook: Gateway to Leadership and Learning; 40

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

15. 16.

17.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

18.

19. 20. 21. 22. 23.

24. 25.

26.

27.

28.

29.

30. 31.

Kochan, F., Searby, L., Barakat, M., Eds.; Auburn University: Auburn, AL, 2012. Klemm, W. R. Computer slide shows: A trap for bad teaching. Coll. Teach. 2007, 55, 121–124. Manarin, K.; Carey, M.; Rathburn, M.; Ryland, G. Critical Reading in Higher Education Academic Goals and Social Engagement; Indiana University Press: Bloomington, IN, 2015. Lage, M. J.; Platt, G. J.; Treglia, M. Inverting the classroom: A gateway to creating an inclusive learning environment. J. Econ. Educ. 2000, 31, 30–43. Bergmann, J.; Sams, A. Flip Your Classroom: Reach Every Student in Every Class Every Day; International Society for Technology in Education: Arlington, VA, 2012. Daniels, H. Literature Circles: Voice and Choice in Book Clubs and Reading Groups, 2nd ed.; Pembroke Publishers: Markham, ON, Canada, 2002. Shelton-Strong, S. J. Literature circles in ELT. ELT J. 2012, 66, 214–223. Valentino, M. L. Donor funded Open Educational Resources: making the case. The Bottom Line: Managing Library Finances 2015, 28, 112–118. Senack, E.; Donoghue, R. Covering the Cost: Why We Can No Longer Afford To Ignore High Textbook Prices; Student PIRGs: Washington, DC, 2016. Jhangiani, R. S.; Pitt, R.; Hendricks, C.; Key, J.; Lalonde, C. Exploring Faculty Use of Open Educational Resources at British Columbia Post-Secondary Institutions; BCcampus Research Report; BCcampus: Victoria, British Columbia, Canada, 2016. ChemWiki. ChemWiki: The Dynamic Chemistry Hypertext, 2016. http:// chemwiki.ucdavis.edu/ (accessed March 23, 2016). Rusay, R. J.; Mccombs, M. R.; Barkovich, M. J.; Larsen, D. S. Enhancing undergraduate chemistry education with the online dynamic ChemWiki resource. J. Chem. Educ. 2011, 88, 840. Halpern, J. Why the ChemWiki? ACS CHED CCCE Newsletter,2015, Fall, Paper 7. http://confchem.ccce.divched.org/2015FallCCCENLP7 (accessed March 23, 2016). Allen, G.; Guzman-Alvarez, A.; Smith, A.; Gamage, A.; Molinaro, M.; Larsen, D. S. Evaluating the effectiveness of the open-access ChemWiki resource as a replacement for traditional general chemistry textbooks. Chem. Educ. Res. Pract. 2015, 16, 939–948. Morsche, L. In The Flipped Classroom; Muzyka, J. Ed.; American Chemical Society Symposium Series 1223 and 1228; Amreican Chemical Society: Washington, DC, 2016. McCollum, B. M. Chem 1201, 2016. http://chemwiki.ucdavis.edu/ Wikitexts/Mount_Royal_University/Chem_1201 (accessed March 23, 2016). Rhodes, A.; Rozell, T. A constructivist approach to e-text design for use in undergraduate physiology courses. Adv. Physiol. Educ. 2015, 39, 172–180. Straumsheim, C. No Rush to ‘Go Digital’, February 22, 2016. Inside Higher Ed. https://www.insidehighered.com/news/2016/02/22/study-facultymembers-skeptical-digital-course-materials-unfamiliar-oer (accessed March 23, 2016). 41

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

32. Freund, L.; Kopak, R.; O’Brien, H. The effects of textual environment on reading comprehension: Implications for searching as learning. J. Info. Sci. 2016, 42, 79–93. 33. Margolin, S.; Driscoll, C.; Toland, M.; Kegler, J. E-readers, computer screens, or paper: Does reading comprehension change across media platforms? Appl. Cogn. Psychol. 2013, 27, 512–519. 34. DeStefano, D.; LeFevre, J.-A. Cognitive load in hypertext reading: A review. Comput. Human Behav. 2007, 23, 1616–1641. 35. Zambarbieri, D.; Carniglia, E. Eye movement analysis of reading from computer displays, eReaders and printed books. Ophthalmic Physiol. Opt. 2012, 32, 390–396. 36. Kretzschmar, F.; Pleimling, D.; Hosemann, J.; Fussel, S.; BornkesselSchlesewsky, I.; Schlesewsky, M. Subjective impressions do not mirror online reading effort: Concurrent EEG-eyetracking evidence from the reading of books and digital media. PLoS ONE 2013, 8, e56178. 37. Rockinson- Szapkiw, A.; Courduff, J.; Carter, K.; Bennett, D. Electronic versus traditional print textbooks: A comparison study on the influence of university students’ learning. Comput. Educ. 2013, 63, 259–266. 38. McCollum, B. M. In Teaching Science Online; Kennepohl, D., Ed.; Stylus: Winnipeg, Manitoba, Canada, 2016. 39. Nickle, T. Mount Royal University, Calgary, Alberta, Canada. Personal communication, October 28, 2015. 40. Abeysekera, L.; Dawson, P. Motivation and cognitive load in the flipped classroom: definition, rationale and a call for research. High. Educ. Res. Dev. 2015, 34, 1–14. 41. Saitta, E.; Morrison, B.; Waldrop, J. B.; Bowdon, M. A. In Best Practices for Flipping the College Classroom; Waldrop, J. B.; Bowdon, M. A., Eds.; Routledge: New York, 2016; pp 1–16. 42. Gosser, D.; Roth, V. The Workshop Project: Peer-led Team Learning. J. Chem. Ed. 1998, 75, 185–187. 43. Mazur E. Peer Instruction: A User’s Manual; Prentice Hall: Upper Saddle River, NJ, 1997. 44. Novak, G.; Patterson, E. T. The best of both worlds: WWW enhanced in-class instruction. Proceedings of the IASTED International Conference on Computers and Advanced Technology in Education; ACTA Press: Calgary, Alberta, Canada, 2000; pp 1–7. http://serc.carleton.edu/resources/ 14236.html (accessed March 23, 2016). 45. Moravec, M.; Williams, A.; Aguilar-Roca, N.; O’Dowd, D. K. Learn before lecture: a strategy that improves learning outcomes in a large introductory biology class. CBE-Life Sci. Educ. 2010, 9, 473–481. 46. Bowen, J. A. Teaching naked: How moving technology out of your college classroom will improve student learning; Jossey-Bass: San Francisco, CA, 2012. 47. Shibley, I.; Amaral, K. E.; Shank, J. D.; Shibley, L. R. Designing a blended course: using ADDIE to guide instructional design. J. Coll. Sci. Teach. 2011, 40, 80–85. 42 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

48. Seery, M. K.; Donnelly, R. The implementation of prelecture resources to reduce in-class cognitive load: a case study for higher education chemistry. Brit. J. Educ. Technol. 2012, 43, 667–677. 49. Smith, J. D. Student attitudes toward flipping the general chemistry classroom. Chem. Educ. Res. Pract. 2013, 14, 607–614. 50. Christiansen, M. A. Inverted teaching: applying a new pedagogy to a university organic chemistry class. J. Chem. Educ. 2014, 91, 1845–1850. 51. Yeung, K.; O’Malley, P. J. Making ‘the flip’ work: barriers to and implementation strategies for introducing flipped teaching methods into traditional higher education courses. New Dir. 2014, 10, 59–63. 52. Fautch, J. M. The flipped classroom for teaching organic chemistry in small classes: is it effective? Chem. Educ. Res. Pract. 2015, 16, 179–186. 53. Seery, M. K. Flipped learning in higher education chemistry: emerging trends and potential directions. Chem. Educ. Res. Pract. 2015, 16, 758–768. 54. Rein, K. S.; Brookes, D. T. Student response to a partial inversion of an organic chemistry course for nonchemistry majors. J. Chem. Educ. 2015, 92, 797–802. 55. Flynn, A. B. Structure and evaluation of flipped chemistry courses: organic and spectroscopy, large and small, first to third year, English and French. Chem. Educ. Res. Pract. 2015, 14, 198–211. 56. Hibbard, L.; Sung, S.; Wells, B. Examining the effectiveness of a semi-selfpaced flipped learning format in a college general chemistry sequence. J. Chem. Educ. 2016, 93, 24–30. 57. Yestrebsky, C. In Best Practices for Flipping the College Classroom; Waldrop, J. B.; Bowdon, M. A., Eds.; Routledge: New York, 2016; pp 17–28. 58. Eichler, J. F.; Peeples, J. Flipped classroom modules for large enrollment general chemistry courses: a low barrier approach to increase active learning and improve student grades. Chem. Educ. Res. Pract. 2016, 17, 197–208. 59. Bradley, A. Z.; Ulrich, S. M.; Jones, M., Jr.; Jones, S. M. Teaching the sophomore organic course without a lecture. Are you crazy? J. Chem. Educ. 2002, 79, 514–519. 60. Kalyuga, S.; Ayres, P.; Chandler, P.; Sweller, J. Expertise reversal effect. Educ. Psychol. 2003, 38, 23–31. 61. Kalyuga, S. In The Cambridge Handbook of Multimedia Learning; Mayer, R., Ed.; Cambridge University Press: New York, 2005; pp 325–337. 62. Toledo Examination; Division of Chemical Education Examinations Institute, American Chemical Society: Ames, IA, 2009. 63. Ryan, M. D.; Reid, S. A. Impact of the flipped classroom on student performance and retention: A parallel controlled study in general chemistry. J. Chem. Educ. 2016, 93, 13–23. 64. Herreid, C. F.; Schiller, N. A. Case studies and the flipped classroom. J. Coll. Sci. Teach. 2013, 42, 62–66. 65. Poh, M.-Z.; Swenson, N. C.; Picard, R. W. A wearable sensor for unobtrusive, long-term assessment of electrodermal activity. IEEE Trans. Biomed. Eng. 2010, 57, 1243–1252. 43 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

66. McCollum, B. M. In Exploring the Role of Instructional Styles on Learning Experiences in a Technology-Enhanced Classroom with Open Educational Resources; 2015 Symposium on Scholarship of Teaching and Learning, Banff, Alberta, Canada, November 12–14, 2015; Miller-Young, J., MacMillan, M., Rathburn, M., Eds.; Institute for Scholarship of Teaching and Learning: Calgary, 2015. 67. Jensen, J. L.; Kummer, T. A.; Godoy, P. D. d. M. Improvements from a flipped classroom may simply be the fruits of active learning. CBE-Life Sci. Educ. 2015, 14, 1–12. 68. Kim, M. Literature discussions in adult L2 learning. Lang. Educ. 2004, 18, 145–166. 69. McElvain, C. M. Transactional literature circles and the reading comprehension in English learners in the mainstream classroom. J. Res. Read. 2010, 33, 178–205. 70. Seburn, T. Academic Reading Circles; The Round: Toronto, Ontario, Canada, 2015. 71. Stahl, N. A.; King, J. R.; Eiler, U. Postsecondary reading strategies rediscovered. J. Adolesc. Adult Lit. 1996, 39, 368–379. 72. Robinson, F. P. Effective Study; Harper: New York, 1946. As referenced in Stahl, N. A.; King, J. R.; Eiler, U. Postsecondary reading strategies rediscovered. J. Adolesc. Adult Lit. 1996, 39, 368–379. 73. Graham, S.; Hebert, M. Writing to read: A meta-analysis of the impact of writing and writing instruction on reading. Harvard Educ. Rev. 2011, 81, 710–744. 74. Lever, K. M. Mobile Music Technology, Communication Isolation and Community Building: An Analysis of College Students’ Use of Digital Entertainment. Ph.D. Thesis, Graduate School, New Brunswick Rutgers, NJ, 2007. http://dx.doi.org/doi:10.7282/T3P84C95 (accessed March 23, 2016). 75. Kodama, C. Marginality of transfer commuter students. J. Stud. Aff. Res. Pract. 2002, 39, 248–265. 76. Funk, M. T. Assessing nontraditional student dropouts on a commuter campus. Dissertations, 2015, Paper 80. 77. Grisham, D. L.; Wolsey, T. D. Recentering the middle school classroom as a vibrant learning community: Students, literacy, and technology intersect. J. Adolesc. Adult Lit. 2006, 49, 648–660. 78. Williams, C. Learning On-line: A review of recent literature in a rapidly expanding field. J. Furth. High. Educ. 2002, 26, 263–272. 79. Seufert, T.; Jänen, I.; Brünken, R. The impact of intrinsic cognitive load on the effectiveness of graphical help for coherence formation. Comput. Human Behav. 2007, 23, 1055–1071. 80. McCollum, B. M.; Regier, L.; Leong, J.; Simpson, S.; Sterner, S. The Effects of Using Touch-Screen Devices on Students’ Molecular Visualization and Representational Competence Skills. J. Chem. Educ. 2014, 91, 1810–1817. 81. McCollum, B. M.; Sepulveda, A.; Moreno, Y. Representational technologies and learner problem solving strategies in chemistry. Teach. Learn. Inquiry 2016, 4 (4). 44 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch002

82. Gordon, M. Between constructivism and connectedness. J. Teach. Educ. 2008, 59, 322–331. 83. Parr, C.; Woloshyn, V. Reading comprehension strategy instruction in a firstyear course: An instructor’s self-study. Can. J. SoTL. 2013, 4, Article 3. 84. Butt, A. Student views on the use of a flipped classroom approach: evidence from Australia. Bus. Educ. Accred. 2014, 6, 33–43. 85. Michaelsen, L. K.; Sweet, M. The essential elements of team-based learning. New Dir. Teach. Learn. 2008, 116, 7–27. 86. Sweet, M.; Pelton-Sweet, L. M. The social foundation of team-based learning: Students Accountable to Students. New Dir. Teach. Learn. 2008, 116, 29–40. 87. Bennett, L. Putting in more: emotional work in adopting online tools in teaching and learning practices. Teach. High. Educ. 2014, 19, 919–930.

45 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

Chapter 3

Using Web 2.0 Technology in Assessment of Learning in Chemistry: Drawing Threads between Teaching as Practice and Teaching as Research Gwendolyn A. Lawrie* School of Chemistry & Molecular Biosciences, The University of Queensland, St. Lucia, Queensland 4072, Australia *E-mail: [email protected]

Chemistry academics are increasingly encouraged to adopt new technologies both to engage their students in active learning and to assess their learning outcomes. In parallel, they are faced with a shifting landscape in terms of learning environments in tertiary education that includes blended learning and new teaching spaces. For many however, the value of the technology is not always immediately evident and while careful pedagogical strategies can enhance learning and assessment through use of learning technologies or new media, successful implementation is not guaranteed. In this chapter, the synergy between teaching as practice and teaching as research is explored with examples of assessment that have been enhanced using web 2.0 technologies as the vehicle. The affordances, limitations and barriers of these technologies are explored to establish recommendations in terms of teaching practice and evaluating student learning outcomes.

© 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

Introduction Chemistry teaching is constantly being transformed through hybrid new learning environments where teachers are encouraged to support student learning using multiple technologies and new media. To enable students to participate in active learning environments, their teachers need pedagogical knowledge and skills to orchestrate higher impact learning activities. In clarification of the terminology used in this chapter, the ‘teachers’ referred to are tertiary chemistry faculty (also termed ‘academics’ in international contexts) who are teaching in higher education settings (universities and colleges). In this context, reference to ‘assessment’ is the practice of measuring of student learning progress and ‘evaluation’ is the process of measuring the effectiveness of teaching and assessment practices. Taking the first steps towards using technology in teaching and assessment may seem daunting to many chemistry faculty. Most are unfamiliar with the complexity involved in designing learning activities that are facilitated by technologies, nor do they have the required pedagogical knowledge and skills. It is unlikely that tertiary chemistry teachers will be familiar with the technological pedagogical and content knowledge (TPACK) framework that attempts to make the dynamic relationships between teaching, learning and technology explicit (1–3), this framework has arisen from the context of STEM high school teacher preparation (4). A complicating factor for chemistry faculty who are seeking to be informed by pedagogies and practices that involve instructional technologies, is that technology-based educational research derives from diverse fields and is published in their associated journals including: disciplinary-based education; higher education; computer-supported education; assessment; and educational psychology journals. Authorship teams will often include stakeholders from across the tertiary sector including both teaching faculty and academic developers, a large number of the latter have transferred from their discipline of training into educational research. Grainne Conole (University of Leicester) identifies as a tertiary chemistry academic and is a leading researcher in the field of theory, evaluation and practice of learning innovation and design using technologies and digital media to enhance student learning. In the preface to her book about designing for learning, which is a good starting place to understand the many facets of using technologies in learning, she recognizes that for many faculty, external factors are pushing them into the realm of adopting technology tools (5): ‘This is an exciting time in education, which is operating within an increasingly complex societal context, one of rapidly changing technologies and increasing financial constraints. New social and participatory media have much to offer for learning and teaching. To address this challenging context, we need to radically rethink the way in which we design, deliver, support and assess learning.’ While substantial knowledge and guidance already exists in the use of technologies and digital media in learning and teaching, it is typically beyond 48 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

the capacity of most tertiary chemistry teachers to monitor, and filter, this broad body of contemporary literature. It also takes substantial time to translate generalized findings and recommendations from this research base into their own chemistry teaching contexts. On a positive note, publication of discipline-based e-learning research studies is becoming more widespread, particularly in terms of pedagogies that are enhanced through technology (6, 7). Indeed, readers will find that each issue of most discipline-based education journals in STEM (for example the Journal of Chemical Education) contains many reports of the adaptation of teaching practice through technology across learning environments including online learning, the laboratory and lectures. Digital media represents a versatile suite of tools that serves as a platform from which to facilitate and assess student created learning artefacts either online or through institutional learning management systems (LMS). The introduction of web 2.0 tools to enhance learning requires careful consideration of the intended learning outcomes for students – the rationale for using technology to enhance learning or assessment must be driven by making the technology almost ‘invisible’ in the processes of learning. Adopting technology just because it is what students are using, or because it has been prescribed as an institutional directive (top down), will potentially result in unsuccessful implementation or disenfranchised students because they will not connect to its purpose (8). It is therefore important to match the affordance of the technology to measuring individual student learning in chemistry while enabling them to value its purpose – the challenge is to capture individual student’s chemistry thinking. Chemists rely on multiple external representations to support depiction or explanation of chemical models, processes and concepts (9–11). Therefore, to learn and become proficient in chemistry, students must acquire, apply and translate between multiple symbols, structures, terminology and representations that are inherent to the discipline. Students across all levels of their learning often view learning chemistry as difficult - there have been multiple studies to explore why they have this perception. One of the most useful and persistent models that has arisen informed by research explains the relationship between working memory and cognitive load: the information processing model (readers are directed to Reid’s in-depth review of the basis of this model as a starting point) (12). The key elements of the information processing model are the students’ ability to filter information from external sources, process the information in the working memory to apply it and then transfer inter-related ideas between the working memory and long-term memory. This model has been graphically modified as the basis of this chapter to illustrate the points at which assessment of student learning outcomes may be accessed using technology (Figure 1). It is clear from Figure 1 that assessment of student learning can be completed at different stages of learning and may illicit different levels of student understanding. The specific point of interest in this chapter is assessment of student-created explanations and representations through the application of technology which may provide deeper insight into students’ understanding. The journey then begins into the realms of trialling a new technology in the assessment of chemistry learning. This chapter encourages chemistry faculty to ‘dip their 49 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

toes’ into evidencing their technology-enhanced teaching practices that have been informed by research (scholarly teaching) (13). Scholarly teaching is independent of discipline and generally begins as informed practice as a result of reading educational literature and reflection – for many faculty, the dissemination of a teaching innovation never leaves the classroom or is only framed in conversations with colleagues in the same institutional department. The subsequent steps towards ‘teaching as research’ for chemistry faculty involves the identification of strategies to collect evidence of student learning in response to a change in pedagogical practice (13). Many chemistry faculty will engage in action research (13) where they implement an innovation into their own teaching context, reflect on the outcomes and make further changes as required. Extension of scholarly teaching practice into teaching as research may be achieved through use of theoretical frameworks to inform learning design and to add a more rigorous evaluation (including institutional ethical approval if individual student data is to be disseminated). The key differences between the two can be framed through a series of reflective questions (Figure 2).

Figure 1. The elements of the information processing model for learning chemistry translated into a model for discriminating assessment. (a) Assessment of working memory as concepts are encountered and applied in problem solving. (b) Assessment of students’ ability to transfer and apply concepts in new contexts. (c) Assessment of application of concepts after a period of time. (d) Assessment of students ability to explain concepts to an audience.

The underlying philosophy of academic research means that only successful implementations are ever published. In reality, in the context of teaching and learning research, it would be extremely informative if authors shared insights into their unsuccessful strategies as part of their teaching journeys to successful practices. In this chapter, several examples of designing for learning involving 50 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

technology to enhance assessment are shared including strategies that did not work. These studies incorporated theoretical and evaluation frameworks to inform the design for learning hence the effectiveness of each initiatives was evidenced to provide insights into the affordances and limitations of technology.

Figure 2. Key differences between scholarly teaching and teaching as research.

Using Technology To Assess Learning in Chemistry An overwhelming array of technologies and tools exists that could be applied to assess a student’s learning at the different points in Figure 1. Over several decades, each evolution in the technologies available to enhance learning has spawned a new wave of pedagogies and practices – however, many of the barriers and challenges appear to have remained the same (14, 15). Indeed a major focus of educational research has been the prior experiences and skills of both teachers and their students. These collective experiences and skills are referred to individually, interchangeably or collectively as information, communication technology (ICT) literacy, digital literacies and/or technological skills (5, 16, 17). The spectrum of the research into the use of technology to enhance learning and assessment has ranged from the development of theoretical framworks, learning design strategies and evaluation methods (5, 17) to the exploration of the utility of a specific web 2.0 tool in a teaching and learning, for example Youtube (18, 19). At an even more granular level, there are numerous studies that communicate the outcomes of the implementation of a single tool into a specific context and in more recent years embedded theoretical frameworks are becoming more prevalent (20, 21). In parallel, there has been extensive debate as to whether students are ‘Digital Natives’ and how that translates into their learning, self-efficacy and autonomy (8, 22–26). The term digital natives is meant to characterize the gap between students who have grown up surrounded by technology and their teachers, who are rapidly upskilling to deliver learning experiences enabled through technology 51 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

(sometimes referred to as ‘digital immigrants’) (8). In reality, there are substantial differences in students’ technological skills based on a range of factors including gender and sociocultural factors (22, 26, 27). Teachers should be aware of these factors, and take them into account, when designing activities for learning for their own contexts and students.

Figure 3. Multiple dimensions representing the interplay between knowledge, experience and skills of teachers (technological pedagogical and content knowledge, TPACK, and topic specific professional knowledge, TSPK) and their students in the implementation of new media and web-based technologies for learning and assessment.

As part of designing the learning activity or assessment task, chemistry faculty will draw upon their topic specific professional knowledge (TSPK) (28) of how to teach a concept. They will need to possess or acquire technological skills to implement the task (troubleshooting issues with browsers and system permissions) and scaffold their students learning. This relies on their pedagogical content knowledge in the utility of technologies (TPACK, Figure 3). Their students will likely possess a wide range of prior experiences and competencies in social media, technological skills and software skills – these are collectively attributed as digital literacies. However, in Figure 3, the different domains broadly cluster skills and proficiencies in terms of activities where information and communication technology literacy involves the use of the internet to seek, retrieve, filter and communicate digital information including social media and 52 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

collaboration tools. Technological skills represent students’ abilities in managing their devices to access or create digital objects including software or application updates, managing accounts and troubleshooting access. Finally, software skills include using standard applications such as Microsoft Office programs and saving or converting between file formats. The tertiary teacher will need to evaluate, identify and lower any barriers that arise during the implementation of technology in learning and assessment. Teachers are able to identify these issues through knowledge of their own students over time and this contributes towards the teacher’s TSPK (28). In tertiary chemistry, this becomes challenging as teachers are often faced with very large classe sizes and may only know a student for one or two semesters.

Mapping Web 2.0 Tools against Assessment of Intended Learning Outcomes When selecting the appropriate web 2.0 tool to facilitate an assessment task, the affordances and limitations of the tool must be weighed up against the intended student learning outcomes to identify the net benefits that will be evident to both the teacher and their students. A second consideration is whether students will be working as individuals or with their peers in the task. Many web 2.0 tools can facilitate collaborative knowledge construction and networked communities – these require careful learning design to maintain learner autonomy and creativity (29). There have been recent attempts to map different web 2.0 technologies to assessment of learning outcomes by aligning with learning taxonomies (Blooms taxonomy (30) or SOLO taxonomy (31)). The simplest approach is to consider what the assessment is measuring in terms of student learning (Figure 1) and then mapping this to the affordances of a technology (Table 1). Evaluation frameworks provide the greatest structure to the processes of identifying research questions and collecting data that can inform the outcomes of a research study. One of the most versatile frameworks for evaluating the learning design and the efficacy of technology implementation to enhance learning and assessment is the LEPO framework. Developed by Phillips, McNaught & Kennedy, this framework is designed to consider the components of the learning environment, learning processes and learning outcomes (LEPO) as a result of intentional learning design (32).

53 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

Table 1. Examples of Web 2.0 tools, the affordances that enable technology-enhanced assessment and alignment with Figure 1 Web 2.0 technology/tool

Affordances

Assessment

Example tasks

Video (e.g. Youtube or other video sharing site)

Representations and explanations provide insight into understanding and mental models. Shared products of group work

Explanation of concepts to an audience

Vlog reflections, Video communication of problemsolving outcomes

Audio or screen capture (e.g. Camtasia)

Verbal explanations captured synchronously with representations

Explanation of concepts to an audience

Peer tutorial

Blog or webpage (e.g. Wordpress)

Reflection on processes and change in practice over time

Transfer and application of concepts in new contexts

Problemsolving, laboratory report

Wiki (e.g. PBWorks)

Collaborative co-construction of understanding

Explanation of concepts to an audience

eLab Notebook

Social Media (e.g.Twitter or Facebook)

Observations in experiments or collation of examples

Communication of an idea

eportfolio

Concept mapping applications (e.g. Mindmeister)

Asynchronous representation of students understanding

Transfer and application of concepts to show connectivity

Concept map

Shared fora (e.g.Facebook Groups, Google Docs)

Peer discussion to reach a collective consensus

Explanation of concepts to an audience

Study Group, collaborative writing

Examples of Technology-Enhanced Assessment in Chemistry Learning Several examples of the adaptation of web 2.0 tools to enhance assessment of learning in chemistry in a single university are described below to make explicit the tensions that may arise in finding a balance between ‘teaching as practice’ and ‘teaching as research’. Each journey in the introduction of technology to measure student learning in chemistry had several dimensions. The rationale, the design for learning, technology tool and pedagogical strategies are summarized in Table 2. The successful outcomes of each have been published elsewhere and are not revisited in-depth here, instead the key findings along with the trials that were unsuccessful during implementation are shared.

54 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

Table 2. The rationale, evaluation and dissemination of assessment involving applications of specific web 2.0 technologies Instructional design

Video blogs

eLab notebook

Collaborative inquiry group task

Personal element tweet

Rationale

Use visual aids and video explanations to engage students in learning chemistry structures and representations

Evidence a wiki collaborative domain as supporting co-construction of shared understanding and scientific literacy

Introduce a collaborative small group inquiry task where students are interdependent on each other and embed peer review and assessment

Use social media to increase the personal relevance of chemistry

Thread (Figure 2)

Scholarly teaching that evolved into teaching as research

Teaching as research

Teaching as research

Scholarly teaching

Context

Introductory Chemistry

Upper division Nanoscience

General Chemistry

Introductory Chemistry

Class size

360

30

1400+

120

Increase individual accountability and learning in group research based labs

Increase the relevance of chemistry and develop transferable skills

Introduce a personal connection to chemistry and use social media in teaching

Aim

Web 2.0 tool

Youtube

PBWorks

iCAS bespoke web-based collaborative group tool

Twitter/ Blackboard wiki

Affordance

Synchronous explanations and interactions with visual aids.

Groups have a dedicated wiki and can co-create content. Instructor can view histories

Management of large numbers of students through small group work and peer assessment

Shared ‘Tweets’ about students’ personal elements

Theoretical framework

Adaptation of Pierce’s semiotic triad (33)

Conversational framework (34)

Cooperative groupwork and interdependency (35)

None

Continued on next page.

55 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

Table 2. (Continued). The rationale, evaluation and dissemination of assessment involving applications of specific web 2.0 technologies Instructional design

Video blogs

eLab notebook

Collaborative inquiry group task

Personal element tweet

Evaluation

Qualitative & quantitative data analysed by themes sourced in literature

LEPO framework used to collect qualitative & quantitative data

LEPO framework used to collect qualitative & quantitative data

Institutional course evaluation

Dissemination

Conferences, journal article (36) and book chapter (37)

Conferences, journal article (21) and book chapter (20)

Conferences, journal article (35) and teaching resource (38)

None

Video Blogs (vlogs) The rationale for the introduction of this assessment task into an introductory chemistry course was to engage students in translating between and explaining representations of chemical structures (36). They were required to choose a molecule or substance that had strong personal relevance then explain how its structure gave it the function or properties that made it important to them. The assessment was targeted at increasing and capturing students’ understanding of chemical concepts at point (d) in Figure 1 through their creation of video explanations for an audience. The web 2.0 platform of video share websites: Youtube or Vimeo was adopted to facilitate submission of these student created videos which are often extremely large files (full recommendations for practice have been published elsewhere) (37). The affordances provided by these video-sharing websites included: easy instructor accessibility; student retention of ownership of their product (with privacy settings); time-stamped upload information; options for peer review and assessment; and enhanced technological literacies. While, the learning design and implementation of this task was successful, two significant skills-related challenges arose during the process and included the wide range of technological skills that students possessed through prior experience and in particular, their use of video editing software (iMovie and MovieMaker). The second challenge was the process of uploading their video files to the video sharing sites and subsequent access for viewing. It was required that students set their videos as restricted to being only viewed by people that they shared the URL link with to protect their privacy (though many deliberately chose to leave their videos as full access). The instructors found many that videos were not initially accessible for marking and they had to contact students to change the privacy settings. There was no possibility of exploitation of this issue by students to gain extra time to work on their videos as they were timestamped on upload. The actual URLs were submitted through an assessment task link in the learning 56 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

management system, Blackboard. These issues were resolved through explicit instructions to scaffold multiple aspects of the task (37). While the original learning design of this task was proven as successful through evaluation, subsequent adaptations to practice revealed several factors that appeared to undermine both the learning design and the learning outcomes for students. An example was when, in response to significantly increased class enrollments creating an increased marking workload, the instructor allowed students to opt to work in pairs on their video submission – in 2012, 17% of students opted to work as pairs (341 students completed the task). While there was no significant statistical difference in the average marks between students who submitted individual tasks (mean = 33.72/40; sd = 4.37) and those who worked as pairs (mean = 34.47/40; sd = 2.77), applying an independent samples test, it was evident that there was a negative impact on the students’ learning experience in pairs. Many pairs of students had divided the task between them which had potential to reduce their individual learning from the task. It was also evident that their stated connection/relevance to their choice of molecule or substance was less personal - this would require a strong connection to the molecule by both students in a pair. They negotiated a substance that was of general relevance to both students in the pair. In subsequent years, the task returned to require individual videos and a team of tutors was engaged in marking the videos.

eLab Notebooks Wikis offer substantial potential for students to work collaboratively in co-construction of a shared written product and to substitute for traditional genres of scientific communication. Students often work in groups in upper level laboratory practicals and when this involves inquiry or research-based experiments, a wiki offers a platform for a laboratory notebook or report (20). The many benefits from adopting this web 2.0 tool include shared information between students, collaboration outside the classroom context, and gains in scientific literacy. The rationale for using the wiki in assessment was to capture student gains in understanding through a collaborative platform where students presented and refined their understanding through explanation (point (d) in Figure 1). There is no doubt that there is a level of ongoing investment of time by an instructor to implement this format of assessment but this is offset by the insights into student thinking that became available through the wiki history function (21). The trade-off between the teacher perspective and the researcher perspective can be very significant for online collaborative environments such as wikis when they are used to enhance face-to-face on-campus activities. From a researcher’s perspective, it would be preferable if individual students worked in isolation to collaborate with their group in the online forum so that the efficacy of the online domain can be evaluated directly. Also, many educational researchers might argue that a class should be divided or compared to a different class to establish a control group. This would enable a comparison to measure the genuine effect of the technology itself on learning processes. In contrast, the teaching practitioner values the wiki environment as an extension of the face-to-face collaborations 57 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

in the laboratory and recognizes that many entries in the wiki will have been discussed by students outside the wiki environment. They also will not exclude students from an activity that they believe is beneficial for student learning hence are unlikely to establish a ‘control group’ of students simply to gain research outcomes. There is thus a tension between research practice and teaching practice - the net outcome has to be the implementation of a methodology that generates evidence of positive gains for students’ learning. Despite this possible tension, our original study of wiki elab-notebooks was conducted in 2011 (21) and a subsequent evaluation of the 2012 iteration also collected under the same methodology demonstrated that students identified the same themes as benefits of the wiki environments to extend their relationships with their group members, share and process data (Table 3). In line with the original finding of the research into the utility of a wiki for this assessment (21), there is evidence that students are integrating their use of the wiki with other modes of interaction to support their communication of laboratory data (Table 3).

Table 3. Themes that recurred in student feedback collected during the second iteration of the eLab Notebook wikis (previously unpublished data) Theme

Example comment

Shared understanding

“The wiki was great in keeping us all using the same updated data. Instead of multiple group members wasting time doing the same things to the wiki” (S3) “If we had to use the traditional paper notebook etc. either one person would have been more responsible for keeping the record up to date or we would have had to wait and take it in turns to add components to the final collection of data, comments, results and discussion” (S5)

Skills gained from using wiki

“essentially the same as any other notebook/lab book. There was no real increase in ‘technology’ skills, but it did increase my ability to summarise/organise chemistry prac materials” (S1) “easy enough to use once we were given the workshop on it. Didn’t really impact on my technology skills but helped my communication skills” (S4) Continued on next page.

58 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Table 3. (Continued). Themes that recurred in student feedback collected during the second iteration of the eLab Notebook wikis (previously unpublished data) Theme

Example comment

Collaborative relationships

“got to stay in touch with them easily in between labs, ..... made it easier to ‘bond’ with the other group members” (S1)

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

“we were always contacting each other to discuss a new idea. It made our work more coherent” (S2) “it got us to interact better, in comparison to normal lab work where each member would be given their responsibilities, with little interaction between each others responsibilities” (S3) Blending of activity in and out of wiki

“We used facebook a bit to discuss things, since it was easier, and then would either copy the discussion over or summarise it” (S1) “As the experiment came to an end ..... data analysis and discussion became more important and easier to do if we were present together in a group as we added things to the wiki” (S5)

iCAS: A Bespoke Web-Based Technology Tool To Support the Sharing of the Products of Collaborative Group Work and Peer Assessment Whilst wikis can enable instructors to gain insights into student collaborative and communication processes, it is difficult to assess the peer review of the associated collaborative group work products for other groups. Here we define peer review as the critical appraisal of the work of other student groups by an individual student (39). Peer review has potential to capture and assess student learning at point (d) on Figure 1, however it requires careful integration with the other process elements of cooperative small group work to be effective (35). It is difficult to gain insight into any aspect of individual student accountability, participation, communication and learning during small cooperative groupwork tasks which often appear as a ‘black box’. Peer review of the products generated by other groups adds the dimension of feedback and reflection on collective explanations and outcomes. For large classes, this process must be managed through technology and often the technology itself presents barriers to students and instructors alike. We found that our platform (iCAS, interactive collaborative assessment system) developed within our institution, hence is bespoke, had an inherent issue when students attempted to open and peer review the work of their peers. Student groups were required to submit a pdf file for their report through iCAS to provide access to other students. Many groups were unable to convert their text documents into pdf files (35), word processing file formats are dependent on the platform (PC or Mac) and on the version of the software. A second unexpected hurdle presented itself dependent on the names that students had assigned their group’s document electronic file. Some were simply too long! 59 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

With four students in each group, they were tempted to include all their names and student ids into the document name. This issue took substantial time to first identify then resolve – it involved renaming the troublesome files manually. A second very interesting issue arose related to individual student’s submission of written text into the online comment fields for their peer review. Many students believed that their peers would read their entries so they composed comments according to their perceived audience and used punctuation such as apostrophes, exclamation marks or smiley faces. This introduced the occurence of an unexpected html or javascript error which prevented other students entering text for the same student or report, for example an apostrophe in ‘Bryn’s’ is substituted by ‘Bryn's’. It is known that students are able to discriminate between social and formal written text in the appropriate language to use in assessment (40). However, it was likely that students introduced more informal, social language and emoticons to their peer comments based on the audience of their peers – this situation represents a dissonance between the academic design of learning technology systems for assessment and socio-cultural behaviors, and this could be explored further. After the first occurrence of these issues, additional task scaffolding was introduced and the programmers remediated the technology by adding a ‘fix’ that is more tolerant of the text that is entered. This example of the introduction of technology to facilitate assessment highlighted the techological skills that were required of both the chemistry faculty and their students. Since then, many cooperative small group tasks have been successfully implemented applying the design framework and iCAS technology to facilitate the tasks. The group products have included lecture note summaries and most recently group video explanations of the rationale, assumptions and solutions to open quantitative problems set in contemporary chemistry contexts. These latter examples demonstrate the typical practice of teachers in adapting and translating successful classroom practice into new situations – without always adding the dimension of research and evidencing practices beyond their institution.

Personal Element Tweets (where things go wrong) Finally, an example of where an attempted implementation of a web 2.0 technology failed! There is substantial anecdotal evidence, rarely published because only successful outcomes are published, that simply adopting a tool because it is popular does not in itself guarantee successful learning outcomes. In 2009, an attempt was made to harness Twitter and Tweet-ing in our introductory chemistry course, the rationale at that time was that use of an emerging contemporary social communication platform and familiar to students would be engaging. Students were invited to identify their personal element (this was decided by addition of their date and month of birth to generate an atomic number e.g. 7th October = 10 + 7 = 17, the element is chlorine). The formative assessment task required that they write a ‘Tweet’ to summarise their element’s identity, properties and importance then submit their composition. Very few students 60 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

completed the task (7.5 %), on reflection this could be easily attributed to the poor learning design including: •

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

•

•

The task did not actually use Twitter as the platform because advice was received that students should not be required to create personal Twitter accounts for assessment purposes. Instead they were required to post their ‘Tweet’ to a wiki on the learning management system (Blackboard). This instantly removed the authenticity of using a web 2.0 platform and became a writing task. Students could not connect the purpose of the task to their learning, this poor learning design did not make this explicit or link to any summative assessment. The task had low level cognitive demand only requiring retrieval and summarization of information from the long term memory to the working memory as recall. On reflection, this assessment was directed at point (a) on Figure 1 and might only target understanding in their recall of a discrete pieces of information (facts) rather than deeper understanding of concepts, according to the information processing model.

Despite this poor design, there were indicators of individual student creativity amongst the submissions including a rap, poems and well constructed ‘Tweets’. The task was never repeated because there had been no real integration between personal relevance and learning in chemistry. This example, while not evidenced or researched to any depth is a contextual example of the point made earlier in this chapter where technology should be adopted to enhance learning and not simply for the sake of technology. A metaanalysis of studies between 1990 and 2010 that reported classroom or blended learning use of technology (excluding 100% online and distance learning) found that there was an important role in cognitive support for technology in learning (41). One conclusion was that: ‘it’s the pedagogy not the technology that matters, although more correctly it is the synergistic relationship of these two. Technology makes possible strategies that could not otherwise be implemented’. (p. 284)

Summary and Recommendations for Practice Teachers that embark on the journey of implementing technologies in their practices need a degree of tenacity, flexibility and some time available to invest in the process. However, like many investments, the rewards at the end of the journey may outweigh the cost of the initial process and can be sustained through adaptation or translation. This raises the important question of evidencing the processes and outcomes in student learning – it is frustrating that there are probably many instances of a successful adoption of technology in the chemistry classroom that never get disseminated because they weren’t evidenced. This area of chemistry education research and practice is growing quickly but also needs 61 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

many more evidenced studies to demonstrate the role that technology can play in enhancing learning outcomes. For many academics, the leap from scholarly teaching to teaching as research is too large and too complex. To encourage this transition, the following steps are suggested: •

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

•

Only select a web 2.0 technology because it will add value to an intended learning outcome, either in terms of the teacher’s insight into student learning or in encouraging students to gain deeper understanding of chemical concepts. Introduce some level of evaluation of the learning design and student learning outcomes to decide whether a new pedagogy or assessment task is effective – adoption of an evaluation framework will guide the collection of data and adds the rigour that will make publication feasible.

But even just to begin the journey into scholarly teaching, the novice tertiary chemistry academic is encouraged to ask themselves – ‘How do I know if my students are achieving the intended learning outcomes from this learning activity?’ (Figure 2). The information processing model offers a good structure to enable consideration of how an assessment task may align with the intended learning outcomes of learning design. The representational model (Figure 3) has been introduced to guide discussion of the interplay between the different elements that impact on learning design for the introduction of web 2.0 technologies to enhance learning and assessment practices. Several examples, described above, supported the importance of each element through their emergence in evaluation of the learning designs and included the technological skills, digital literacies and software skills of both the students and their instructors. There was also an example of the unsuitability of one web 2.0 technology based on its poor ‘fit for purpose’ and in-authentic adoption in assessment. This model can guide academics who are perhaps considering embarking on designing learning enhanced by technology - to achieve this in tertiary chemistry, there is an added dimension required in the teacher’s professional knowledge and skills due to the multiple domains that chemists operate in. The examples considered in this chapter focused on the implementation of technology-enhanced assessment to support student explanations and communication to audiences that included their peers and their instructors. Communication skills are becoming increasingly valued as learning outcomes in STEM and web 2.0 technologies offer the affordance of enhancing this skills. In the context of tertiary chemistry education in Australia, this form of assessment addresses in part the chemistry threshold learning outcomes (TLO) standards (42). In particular, TLO 4.1 which states that students will ‘Communicate chemical knowledge by presenting information, articulating arguments and conclusions, in a variety of modes, to diverse audiences, and for a range of purposes.’

62 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Acknowledgments

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

I acknowledge the contributions and support of my collaborator A/Prof Lisbeth Grondahl to our wiki teaching and research projects over many years. The contributions of Emma Bartle, Garry Hoban and Will Rifkin to the dissemination of the activities and outcomes of the Vlog project are also gratefully acknowledged. Ongoing inspiration and wisdom from Gabriela Weaver, Kelly Matthews, Carmel McNaught, Joanne Stewart and Lawrence Gahan that has translated into my teaching research and practice is also acknowledged and valued.

References 1.

Mishra, P.; Koehler, M. J. Technological pedagogical content knowledge. A framework for teacher knowledge. Teach. Coll. Rec. 2006, 108, 1017–54. 2. Pamuk, S.; Ergun, M.; Cakir, R.; Yilmaz, H. B.; Ayas, C. Exploring relationships among TPACK components and development of the TPACK instrument. Educ. Inform. Technol. 2015, 20, 241–63, DOI: 10.1007/s10639-013-9278-4. 3. Bower, M.; Hedberg, J. G.; Kuswara, A. A framework for Web 2.0 learning design. Educ. Media Int. 2010, 47, 177–98, DOI: 10.1080/09523987.2010.518811. 4. Niess, M. L. Preparing teachers to teach science and mathematics with technology: Developing a technology pedagogical content knowledge. Teach. Teach. Educ. 2005, 21, 509–23, DOI: 10.1016/j.tate.2005.03.006. 5. Conole, G. Designing for learning in an open world. Explorations in the Learning Sciences, Instructional Systems and Performance Technologies, Volume 4; Springer Science + Business Media, LLC: New York, 2013, DOI: 10.1007/978-1-4419-8517-0. 6. Hoban, G.; Nielsen, W.; Shepherd, A. Student-Generated Digital Media in Science Education: Learning, Explaining and Communicating Content; Taylor and Francis: London, 2015. 7. Seery, M. K.; O’Connor, C. E-learning and blended learning in chemistry education. In Chemistry Education: Best Practices, Innovative Strategies and New Technologies; Garcia-Martinez, J., Serrano-Torregrosa, E., Eds.; Wiley-VCH: Weinheim, Germany, 2015; pp 651–70, ISBN: 978-3-527-33605-0. 8. Bennett, S.; Maton, K. Beyond the ‘digital natives’ debate: Towards a more nuanced understanding of students’ technology experiences. J. Comput Assist. Learn. 2010, 25, 321–331, DOI: doi: 10.1111/j.13652729.2010.00360.x. 9. Hoffmann, R.; Laszlo, R. Representation in chemistry. Angew. Chem. 1991, 30, 1–16, DOI: 10.1002/anie.199100013. 10. Kozma, R. B.; Russell, J. Multimedia and understanding: Expert and novice responses to different representations of chemical phenomena. J. Res. Sci. Teach. 1997, 34, 949–968, DOI: 10.1002/(SICI)10982736(199711)34:93.0.CO;2-U. 63 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

11. Kozma, R.; Russell, J. Students becoming chemists: developing representational competence. In Visualization in Science Education, Models and Modelling in Science Education; Gilbert, J. K., Ed.; Springer: Dordrecht, 2005; pp 121–146. 12. Reid, N. The learning of chemistry: The key role of working memory. In Learning with Understanding in the Chemistry Classroom; Devetak, I., Glazar, S. A., Eds.; Springer: Dordrecht, 2014; pp 77–101, DOI: 10.1007/978-94-007-4366-3_5. 13. Cole, R. Using chemistry education research to inform teaching strategies and design of instructional materials. In Chemistry Education: Best Practices, Innovative Strategies and New Technologies; Garcia-Martinez, J., SerranoTorregrosa, E., Eds.; Wiley-VCH: Weinheim, Germany, 2015; pp 151–80, ISBN: 978-3-527-33605-0. 14. Spotts, T. H.; Bowman, M. A. Increasing faculty use of instructional technology: Barriers and incentives. Educ. Media Int. 1993, 30, 199–204, DOI: 10.1080/0952398930300403. 15. Reid, P. Categories for barriers to adoption of instructional technologies. Educ. Inform. Technol. 2014, 19, 383–407, DOI: 10.1007/s10639-0129222-z. 16. Littlejohn, A.; Beetham, H.; McGill, L. Learning at the digital frontier: A review of digital literacies in theory and in practice. J. Comput. Assist. Learn. 2012, 28, 547–56, DOI: 10.1111/j.1365-2729.2011.00474.x. 17. Beetham, H.; Sharpe, R. Rethinking Pedagogy for a Digital Age: Designing for 21st Century Learning, 2nd ed.; Taylor and Francis: New York, 2013. 18. Snelson, C.; Rice, K.; Wyzard, C. Research priorities for Youtube and video sharing: A Delphi study. Brit. J. Educ. Technol. 2012, 43, 119–29, DOI: 10.1111/j.1467-8535.2010.01168.x. 19. Blonder, R.; Jonatan, M.; Bar-Dov, Z.; Benny, N.; Rap, S.; Sakhnini, S. Can You Tube it? Providing chemistry teachers with technological tools and enhancing their self-efficacy beliefs. Chem. Educ. Res. Prac. 2013, 14, 269–85, DOI: 10.1039/c3rp00001j. 20. Lawrie, G.; Grøndahl, L. Wiki technologies and communities: New approaches to assessing individual and collaborative learning in the chemistry laboratory. In Chemistry Education: Best Practices, Innovative Strategies and New Technologies; Garcia-Martinez, J., Serrano-Torregrosa, E., Eds.; Wiley-VCH: Weinheim, Germany, 2015; pp 671–92; ISBN: 978-3-527-33605-0. 21. Lawrie, G.; Grøndahl, L.; Boman, S.; Andrews, T. Wiki Laboratory Notebooks: Supporting Student Learning in Collaborative Inquiry-Based Laboratory Experiments. J. Sci. Educ. Technol. 2016, 25, 394–409, DOI: 10.1007/s10956-016-9601-0. 22. Hargittai, E. Digital Na(t)ives? Variation in internet skills and uses among the members of the “Net Generation”. Sociol. Inq. 2010, 80, 92–113, DOI: 10.1111/j.1475-682X.2009.00317.x. 23. Margaryan, A.; Littlejohn, A.; Vojt, G. Are digital natives a myth or reality? University students’ use of digital technologies. Comput. Educ. 2011, 56, 429–440, DOI: 10.1016/j.compedu.2010.09.004. 64 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

24. Dabbagh, N.; Kitsantas, A. Personalised learning environments, social media, and self-regulated learning: A natural formula for connecting formal and informal learning. Internet Higher Educ. 2012, 15, 3–8, DOI: 10.1016/j.iheduc.2011.06.002. 25. Prior, D. D.; Mazanov, J.; Meacheam, D.; Heaslip, G.; Hanson, J. Attitude, digital literacy and self-efficacy: Flow-on effects for online learning behavior. Internet Higher Educ. 2016, 29, 91–7, DOI: 10.1016/j.iheduc.2016.01.001. 26. Ting, Y.-L. Tapping into students’ digital literacy and designing negotiated learning to promote learner autonomy. Internet Higher Educ. 2015, 26, 25–32, DOI: 10.1016/j.iheduc.2015.04.003. 27. Yoo, S. J.; Huang, Y-H. D. Comparison of web 2.0 technology acceptance level based on cultural differences. Educ. Technol. Soc. 2011, 14, 241–52. 28. Gess-Newsome, J. A model of teacher professional knowledge and skill including PCK: Results of the thinking from the PCK Summit. In Re-Examining Pedagogical Content Knowledge in Science Education; Berry, A., Friedrichsen, P., Loughran, J., Eds.; Routledge: New York 2015; pp 28–42. 29. McLoughlin, C.; Lee, M. J. W. Mapping the digital terrain: New media and social software as catalysts for pedagogical change. ASCILITE proceedings. In Hello! Where Are You in the Landscape of Educational Technology? Proceedings Ascilite Melbourne, 2008. http://www.ascilite.org/conferences/ melbourne08/progbook/progbook-p1-5.pdf (accessed April 16, 2016). 30. Beetham, H. Designing for active learning in technology-rich contexts. In Rethinking Pedagogy for a Digital Age: Designing for 21st Century Learning, 2nd ed.; Beetham, H., Sharpe, R., Eds.; Taylor and Francis: New York, 2013; pp 94–128. 31. Biggs, J. B.; Collis, K. F. Evaluating the Quality of Learning: The SOLO Taxonomy (structure of the observed learning outcome); Academic Press: New York, 1982. 32. Phillips, R.; McNaught, C.; Kennedy, G. Evaluating e-Learning: Guiding Research and Practice. Routledge: New York, 2012. 33. Hoban, G.; Loughran, J.; Nielsen, W. Slowmation: Preservice elementary teachers representing science knowledge through creating multimodel digital animations. J. Res. Sci. Teach. 2011, 48, 985–1009. 34. Laurillard D. Rethinking University Teaching: A Conversational Framework for the Effective Use of Learning Technologies; Routledge Falmer: London, 2002. 35. Lawrie, G.; Gahan, L. R.; Matthews, K. E.; Weaver, G.; Bailey, C.; Adams, P.; Kavanagh, L.; Long, P.; Taylor, M. Technology supported facilitation and assessment of small group collaborative learning in large 1st year classes: IS-IT Learning? J. Learn. Design 2014, 7, DOI: http://dx.doi.org/10.5204/jld.v7i2.204. 36. Lawrie, G.; Bartle, E. Chemistry Vlogs: a vehicle for student-generated representations and explanations to scaffold their understanding of structure-property relationships. Int. J. Innov. Sci. Math. Educ. 2013, 21, 27–45. 65 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch003

37. Lawrie, G. Video Blogs: A vehicle to support student-generated representations and explanations in chemistry. In Learning Science through Creating Digital Media: Improving Engagement and Understanding; Hoban, G., Nielsen, W., Shepherd, A. ; Routledge: London, 2015; ISBN: 1138833827 38. Lawrie, G.: Gahan, L.; Matthews, K.; Adams, P.; Kavanagh, L.; Long, P.; Weaver, G.; Cusack, R. Handbook of Scenario Resources for Inquiry Learning in STEM: IS-IT Learning? Online Interdisciplinary Scenario-Inquiry Tasks for Active Learning in Large First Year STEM Courses; Australian Learning and Teaching Council (ALTC): Sydney, NSW, Australia, 2011, ISBN: 978-0-642-78123-9. http://www.olt.gov.au/projectonline-interdisciplinary-scenario-inquiry-tasks-uq-2009 (accessed July 15, 2016). 39. Nicol, D.; Thomson, A.; Breslin, C. Rethinking feedback practices in higher education: a peer review perspective. Assess. Eval. High. Educ. 2014, 39, 102–22, DOI: 10.1080/02602938.2013.795518. 40. Grace, A.; Kemp, N.; Martin, F. H.; Parrila, R. Undergraduates’ attitudes to text messaging language use and intrusions of textisms into formal writing. New Media Soc. 2013, 17, 792–809, DOI: 10.1177/1461444813516832. 41. Schmidt, R. F.; Bernard, R. M.; Borokhovski, E.; Tamim, R. M.; Abrami, P. C.; Surkes, M. A.; Wade, C. A.; Woods, J. The effects of technology use in postsecondary education: A meta-analysis of classroom applications. Comput. Educ. 2014, 72, 271–91, DOI: 10.1016/j.compedu.2013.11.002. 42. Buntine, M.; Price, W.; Separovic, F.; Brown, T.; Thwaites, R. Learning and Teaching Academic Standards: Chemistry Threshold Learning Outcome Statements: Australian Learning and Teaching Council, 2011; p 23. http://www.olt.gov.au/system/files/resources/ altc_standards_SCIENCE_240811_v3_0.pdf (accessed April 16, 2016).

66 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Chapter 4

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

Combining Educational Technologies for Student Engagement in the Chemistry Classroom Ginger P. Redd,*,1 Thomas C. Redd,2 Tracie O. Lewis,3 and Etta C. Gravely1 1Department

of Chemistry, North Carolina Agricultural and Technical State University, 1601 East Market Street, Greensboro, North Carolina 27411, United States 2Department of Mathematics, North Carolina Agricultural and Technical State University, 1601 East Market Street, Greensboro, North Carolina 27411, United States 3Instructional Technology Services & Distance Education, North Carolina Agricultural and Technical State University, 1601 East Market Street, Greensboro, North Carolina 27411, United States *E-mail: [email protected]

There is a surge in the implementation of educational technologies for improving student learning in STEM. The use of classroom response systems and content authoring software are examples of the latest technologies for student engagement. Combined, these tools can transform lessons into engaging student-centered, active learning experiences. We present an implementation strategy, in the pedagogical context of improving learning in the chemistry classroom.

Introduction A distinction has been made between using a technology-based tool and truly incorporating technology into the learning cycle in a pedagogical sense (1–4). While technology can be an effective tool to aid in course content delivery (lectures, assignments, etc.), general information exchange (grades, due dates, progress), and organization (assignment submissions, returns, group communications), to influence student learning requires a deeper level of © 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

technology incorporation. In addition, for some of the more recent pedagogical trends towards the use of flipped classrooms and active learning environments, identifying effective technological tools and ways to meaningfully incorporate them into the classroom is essential (5–9). To exploit the benefits of technology in the classroom, a shift towards student centered learning must be made, and its implementation should result in two-way communications, often with real-time feedback for both the student and the instructor. If the thought is followed to conclusion, it suggests a move away from passive learning and towards a cooperative (between teacher and student), collaborative (between students), and active learning based experience (6, 10). During the course of a study on undergraduate STEM reform, it was found that by involving the students more in the learning process, perceived student responsibility was increased, perceived faculty authority was decreased, and the learning process was made more interactive (11, 12). While the general feelings of both faculty and students were positive towards a more interactive, technology based educational experience; several factors can inhibit an instructor’s ability to incorporate technology into their courses. Issues such as a lack of appropriate supporting materials (textbooks, workbooks, etc.), time loss for covering material due to the increased time necessary to facilitate group discussions, and campus paradigms for faculty evaluation and the lack of appropriate tools for the evaluation of faculty using an active learning based delivery system all served as stagnation points in the reform efforts. Globally, there have been many efforts made to identify some of the major obstacles to incorporating more technology into the learning environment, at both the secondary and postsecondary levels. There are significant issues that can affect an institution’s ability to incorporate technology into the learning environment. Based on previous studies, issues of 1) how well a school shifts its focus and expectations to reflect a move towards autonomous learning, 2) availability of infrastructure, 3) availability of training and support personnel for implementation and 4) how well leadership within a school supports the efforts all serve as obstacles which must be addressed to move successfully towards a more technology infused learning paradigm (13–17). As an example, the factors influencing teacher adoption rates were investigated in a Korean study. Not surprisingly, the study found that, while most teachers planned to use technology to support teaching and learning, teachers with little experience were much more likely to incorporate technology into their classrooms voluntarily, while more experienced teachers often had to be compelled to do so. In fact, responding to external requests (such as from administrators or subject area coordinators for the school system) was cited as the primary motivating factor for many of the teachers who infused technology into their classrooms (18, 19). The authors found that it is not enough to incorporate mechanically the use of technology in the classroom (e.g. using it as a mechanism for content delivery), but that the use of technology should be organically infused into the lesson in a way that enhances student learning. A similar study was performed in the Netherlands (13). The study examined the factors impacting educators’ use of information and communication technology (ICT) and their ability to develop innovative applications of it in the 68 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

classroom. While national and local support and mandates were cited as factors that contribute to the success or failure of the initiatives, they were not found to be singularly responsible for any particular outcome on student learning. In addition, factors such as ICT competence and teacher attitudes were found to be necessary but not sufficient for innovative use of ICT. While several conclusions were drawn, of note was the profile of the teacher who was most likely to use ICT successfully in the classroom. The teacher would likely be willing to maintain contact with colleagues and experts in the area, value the advantages of incorporating ICT in the classroom, have a student-oriented approach to education, and have a sufficient ICT competence that is in alignment with the individual’s preferred pedagogical approach. Technological tools that can aid in learning include Classroom Response Systems (such as “clickers”), Content Delivery Systems (such as “SoftChalk”), and Learning Management Systems (such as Blackboard). The tools are not inherently mutually exclusive; there is often overlap in the capabilities of each of the technologies. The Learning Management Systems will often contain both content delivery and assessment capabilities but may not be optimized for immediate student feedback. Similarly, Classroom Response Systems excel at immediate feedback but may only have rudimentary capabilities for assessment and may not provide a deep well of tools for classroom management. More important than the capabilities of the tools is how they can be incorporated in a meaningful, pedagogical sense to aid in student learning in the classroom (4).

Motivations for Incorporating Technological Tools The abundance of technology based tools available and their wide range of capabilities, as compared to the limited resources often available to a department or possibly even an institution as a whole, makes it important to be able to identify what technological tools are most appropriate for a given classroom or classroom objective. Static instruction usually involves little to no interactivity and non-dynamic elements that are typically teacher-centered. The static nature of the lecture delivery involves low-level learning, interaction, or engagement for students (20, 21). Tuovinen (22) identified learner-content interaction, when the student interacts with materials being studied, as the most critical form of interaction. Interactive lessons often contain several elements that encourage student activity and interaction in real time and keep students engaged with the content and instructional materials. Including images and videos in a lesson breaks the monotony of plain, text-based lessons that tend to lose the interest of some learners. Animated images can be used, but with caution in order to avoid distraction from the content (19). Adaptive learning technologies give students follow-up materials based on their performance or response to a designated activity or set of questions (23). They provide real-time updates according to each submission by the student. Thus, allowing the learners to facilitate their mastery of the content. 69 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

Based on a review of several implementations, it is clear that technology use in the classroom should do more than provide a convenient and attractive medium for one-way information transfer. Administrative policy dictates and the development of “best practices” by one faculty member to be used as a blanket approach by all other faculty members or institutions in a system should be avoided as well (18, 24–27). Change for the sake of change relative to instructional practices is not effective and is borderline dangerous. Factors such as teacher training, student demographics, preparation, and cultural differences can all serve as obstacles to a one size fits all approach to teaching and learning (28, 29). Successful incorporation of technology should 1) address directly the beliefs of the individuals involved, 2) involve long-term interventions (longer than a semester) and 3) adapt to the specific institution at which it is being implemented (13, 26).

Location of Study North Carolina Agricultural and Technical State University is an 1890 land-grant university, located in Greensboro, North Carolina, USA. It is a public, doctoral university and is a member of the 17 campus University of North Carolina (UNC) system. At the time of the study, the Department of Chemistry resided in the College of Arts & Sciences (30). Context As of the Fall 2015 semester, the university enrolled 10,852 undergraduate and graduate students making it the largest member of the Historically Black Colleges and Universities (HBCU) with 87.1% of the student population identifying as being a racial minority (31). Approximately, 90.4% of the undergraduate student body are full-time degree seekers and 79.1% of the student body are in-state students. The university offers 55 undergraduate degrees, 31 master’s level degree programs and 9 doctoral programs, including degrees in Chemistry, Chemistry Education, and Chemical Engineering. The Fall 2015 cohort of students included 1,780 first time, full and part-time freshmen. The university employs 411 full time instructional staff with 15 full time instructional staff in the Department of Chemistry (32, 33). The university is ninth on the list of the top 100 baccalaureate producers for African-Americans in Physical Sciences for the 2013-2014 academic year and third on the list for master’s degrees over the same period (34). Three sections of CHEM 104 and three sections of CHEM 106 implemented the engagement activities. CHEM 104 is a general chemistry course designated primarily for students who major in Nursing or Psychology. CHEM 106 is a general chemistry course for students who are majoring in a science, technology, engineering or mathematics (STEM) discipline. The typical student has an SAT math score of 490, or an SAT II math level II score of 470 or and ACT math score of 19 or successful completion of the introductory chemistry course, CHEM 103, with a grade of C or better. 70 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

Both, the CHEM 104 and CHEM 106 courses, have a class size ranging between 50-60 students. Full-time faculty members teach them each in a 3-hour lecture format in an auditorium. The students are typically first or second semester freshmen. The CHEM 106 course is the first course in a two-course sequence. The CHEM 104 course is a terminal course; however, a sophomore may enroll based on the curriculum guide for their major. Due to the course size and the variations in educational preparation, it can be difficult to identify individual student needs and deficiencies. Introducing appropriate technological tools can help to provide instructors with a better sense of each student’s performance, as well as the performance of the class as a whole. The PLRS system developed by Frank Christ (35) offers four steps to the learning cycle: preview, lecture, review and study. In the interest of incorporating technology tools into the learning cycle, the instructors and students follow a model of The Study Cycle adapted from the PLRS system (36). Instructors enhanced The Study Cycle, Figure 1, to increase student engagement within each stage by incorporating educational technology.

Figure 1. Expanded version of The Study Cycle as applied to CHEM 104 and 106, infusing technology tools. Currently, the use of educational technologies is optional at NCAT. In 2013, the course coordinators for CHEM 104 and CHEM 106 adopted textbooks that provided an associated online learning component, MyLabsPlus, and its associated tools. Additionally, in 2014, the University set a mandate that faculty use Blackboard to host course grades, at a minimum. Given established access, it is the learning management system used for the study. We present a discussion of two additional technology based tools and implementation strategies, in the pedagogical context of improving learning in the chemistry classroom 1) via real-time assessment and feedback and 2) via both synchronous and asynchronous content delivery. The tools are components of a student-centered, active learning environment to enhance student comprehension, interaction and retention. 71 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

SoftChalk at North Carolina A&T State University In 2011, North Carolina A&T State University (NCAT) began using SoftChalk as a tool to create interactive and engaging lectures and instructional materials for online courses. Lectures in both face-to-face courses and online courses used heavily PowerPoint presentations. Presentations from face-to-face courses were uploaded to courses, both online and hybrid, in the Blackboard Learning Management System (LMS) without modification. Instructors reported to instructional technology support staff at NCAT that students were not accessing the lectures in Blackboard. In some cases, the students accessed lectures only moments or days prior to a test. During meetings, faculty expressed interest in products or tools that would encourage the students to access the lectures and review the materials in a timely fashion. Instructors wanted students to be active participants in the learning process and arrive prepared for face-to-face class sessions. SoftChalk was attractive to the instructional technology support staff because it allowed faculty to easily copy content and text from basic word processing documents, a tool that most faculty, staff, and students were comfortable using. SoftChalk allows faculty to add interactive media elements and activities to lessons quickly. These factors made it possible for instructors to enhance otherwise static lessons with images, videos, iFrames, questions, interactive activities, and text poppers, all examples of content enhancements available via SoftChalk. The SoftChalk license for NCAT provides access for all employees, including faculty and staff. A more attractive feature is the integration of SoftChalk with the Blackboard environment at NCAT. Integration addresses the issues associated with students failing to access content in a timely fashion as points or grades earned from activities and questions post automatically to the Blackboard Grade Center. Since the initial implementation, NCAT has invested in a SoftChalk Cloud account that allows faculty to modify and share lessons via the cloud rather than following the traditional process of resending or uploading edited versions of lessons. North Carolina A&T State University currently has over 200 SoftChalk accounts. Both academic and on-academic units create and share lessons, including: course lectures, Banner training, and graduate student orientations and training.

Clickers at North Carolina A&T State University North Carolina A&T State University has used a rapid response system for more than 10 years. However, there was not a standard clicker device for the campus until 2013. Initially, a few units on campus used the Turning Technologies clicker product, but interest and usage declined over time. In 2012, several instructors expressed an interest in using clicker products to increase effective management of their courses. Instructors wanted a method to track student attendance and to assess students’ comprehension of concepts and course content during class sessions. In addition, instructors wanted to encourage more student participation during face-to-face class sessions 72 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

The lack of a standard product for the campus led to visits and communications by numerous vendors regarding different clicker products. In some cases, students purchased different devices for different courses, which became costly for students and made it difficult for staff to support multiple products. Several academic and non-academic offices requested that the University identify a standard clicker product for the campus that prompted a pilot to test and evaluate different options. The Instructional Technology Services and Distance Education Department (ITSDE) selected Turning Technologies and iClicker as the vendors included in the pilot because these vendors provided hardware devices. Support staff determined that using hardware devices would be more reliable because the campus networking infrastructure, which has since been enhanced, could not support the use of multiple mobile devices in certain settings throughout the campus. Therefore, the pilot included vendors with hardware devices available for purchase from the bookstore. Vendors with no hardware devices were excluded from consideration. After instructor training provided by Turning Technologies and iClicker, the faculty conducted a pilot study during the spring 2013 semester. Both vendors provided devices for instructors and students to use in classes and for staff participants to use during meetings and other activities. Pilot participants were divided into two groups. One group used the Turning Technologies clicker and second group used the iClicker device. At the conclusion of the evaluation and testing period, the instructional technology support staff held focus groups to discuss the experiences of pilot participants. Surveys responses recommended the iClicker device as the clicker device for the campus. The University officially implemented use of the iClicker devices during the fall 2013 semester. The ITSDE office provided Instructor kits to faculty, upon request. Students purchased or rented clicker remotes through the University Bookstore. Also, the Blackboard building block for iClicker was installed, which allows students to register the clicker remote to our learning management system. This feature allows instructors to synchronize data and scores from class clicker sessions to the Grade Center of Blackboard. Instructors continue to successfully use iClicker devices and the iClicker mobile application, REEF Polling, in their classes.

Combining Educational Technologies in General Chemistry at North Carolina A&T State University Combining the use of Blackboard, a rapid response system and SoftChalk lessons can lead to a robust arsenal for student engagement in chemistry education. Using the three allows for frequent formative assessments that inform our learners of their knowledge and that aid in adapting instruction for improving upon learning gains. During the Fall 2015 semester, two full-time chemistry faculty members, each teaching three sections of general chemistry opted to work collaboratively in the instruction of CHEM 104 and CHEM 106 by incorporating the use of clickers during in-class sessions and implementing instructor-developed SoftChalk 73 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

lessons while continuing the use of online homework and quiz assignments via MyLabsPlus. Each instructor used the pre-developed clicker questions that accompanied the corresponding textbook adopted for the course. The students enrolled in CHEM 104 did not receive the same clicker questions as those students enrolled in CHEM 106. To increase student engagement, students were encouraged to collaborate on their thoughts through small group discussions prior to submitting their responses through clicker. On average, students responded to five clicker questions per in-class session. Students earned credit for participation whether or not they responded correctly. Specifically, instructors awarded one point per correct response and one additional point for participation if the student responded to at least 75% of the questions within a given session. At the end of each class session, the instructors synchronized and posted students’ scores were to the Grade Center within Blackboard. Additionally, the instructors developed brief SoftChalk lessons that addressed basic information such as key terms, concepts and key equations and relationships specifically focused on dimensional analysis and chemical quantities. Students in both courses, CHEM 104 and 106, were administered the same SoftChalk lessons without any modifications to the content coverage. Instructors used Blackboard to deliver these assignments as preview activities for completion prior to class attendance. In order to increase student participation and interaction with the course content, the instructors granted students unlimited attempts and the ability to rework activities. Subsequently, the student responses within the lessons informed instructors of content requiring reinforcement and elaboration during in-class clicker sessions. While access to MyLabsPlus, clickers and SoftChalk were stated course requirements, there was no way to force students to purchase or access the material. At the end of the fall 2015 semester, students were asked to respond to questions based on their experiences in using clickers, SoftChalk, and MyLabsPlus as well as relative to their use of the Study Cycle. Faculty created, administered and analyzed the survey for this study using Qualtrics software, Versions December 2015 and July 2016 of Qualtrics (Copyright © 2016 Qualtrics) (37). (Qualtrics and all other Qualtrics product or service names are registered trademarks or trademarks of Qualtrics, Provo, UT, USA.)

Observations The survey data has been compiled for all students enrolled in CHEM 104 and CHEM 106. All students actively enrolled received a link to the survey through Blackboard, including students who may have discontinued attendance. A total of 162 students responded to the survey. Approximately 83% of students who responded to the question felt that the use of clickers in the course was either somewhat helpful or very helpful in their understanding of the material, Figure 2.

74 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

Figure 2. Student opinions on the helpfulness of clickers in mastering the materials. Similarly, approximately 82% of students who responded to the question on SoftChalk felt that it was helpful or very helpful to their understanding of the material, Figure 3.

Figure 3. Student opinions on the helpfulness of SoftChalk in mastering the material. While there was some variation between the two categories of “helpful” and “very helpful,” for both the clicker and SoftChalk tools, the number of students who felt that the tools were not helpful was consistent (17% - 18%). The questions did not take in to consideration whether respondents actually purchased access to the clicker tool. 75 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

When asked about the impact of the use of both SoftChalk and clickers on their interest in the course, there was an overwhelmingly positive response, Figure 4.

Figure 4. Students’ opinions on the use of SoftChalk and Clickers increasing their interest during class. While acknowledging that the individual tools are often optimized for one application or method of delivery over another, the students were asked which tool they felt was most helpful in their study process, Figure 5.

Figure 5. A view of the student opinions on the helpfulness of Clickers, SoftChalk and MyLabsPlus in studying. The students cited MyLabsPlus at 62% as being the most helpful tool. The responses seem consistent with expectations of out-of-class studying. When considering the students’ responses, it is important to bear in mind that instructors emphasized pre-class preparation as a necessity for initiating the learning process 76 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

as well as the Preview, Review and Reflect stages of The Study Cycle. While many students indicated that the clickers were a helpful tool for understanding the material, its use was confined to in-class sessions. The interaction of students with the material out-of-class, such as reading the textbook and completing the online assignments, was typically accomplished via MyLabsPlus. In addition to the questions related to their use of the technology, students responded to open-ended questions about their methods of preparing for the course and for assessments within the course. The instructors compiled, normalized, and examined students’ responses, taken in their own words, in determining common themes. In compiling the comments, instructors attempted to normalize verbs tenses, capitalization, and variations of similar words through lemmatization using Qualtrics. Student responses of “my labs plus” or “mylab plus” were edited to be a uniform “MyLabsPlus” and “studying” and “studied” were normalized to “study”. While the tenses were modified when appropriate, different uses of the same word were not altered. As shown in Figure 6, relative to SoftChalk, overall student opinions on its use in the course were a bit mixed, with some students finding that “the explanations of concepts were very helpful” or citing “the video explanation” or “lecture before solving the problems” as being helpful.

Figure 6. Student responses to which areas of the SoftChalk module were most helpful. There were others, however, who felt that “None” of the SoftChalk activities were helpful and one individual felt that it was “cumbersome”, stating that “sometimes I couldn’t tell if I had even finished the assignment”. In responding to the question, students could select all answers that they felt applied. When asked how they prepared for the exams, most students cited the textbook and MyLabsPlus, followed by their notes as the main methods by which they reviewed, Figure 7.

77 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

Figure 7. Student responses when asked how they prepared for exams. The survey allowed students to select all methods that applied. The survey did not ask the students to provide a distinction between an online or electronic book versus a physical textbook. In addition, the students did not specify the method of accessing the electronic book. The MyLabsPlus platform offers an electronic version of the textbook as a web link, though students may have had access through other means. Regardless of method of access, the data suggests a preference for the textbook when it comes to reviewing for the exam. Relative to the preference of videos, the respondents did not specify their origin. In this case, the videos could have originated within MyLabsPlus, and thus be a subset of that category, or they could have originated in other online locations, such as YouTube or Khan Academy. While class notes were also a favored method of reviewing, they came in second to the use of MyLabsPlus for preparing for the exams. A number of students also cited reviewing PowerPoint slides that were accessible through MyLabsPlus. The students mentioned using SoftChalk by name to prepare for their tests but, it was usually in conjunction with another activity (reviewing notes, homework, etc.).

Conclusion There are multiple options in terms of educational technologies that may be used in the chemistry classroom. Technologies were dedicated to offering lessons that are static, interactive or adaptive in an effort to improve upon student learning outcomes. It is important that these tools be leveraged to offer chemistry students a robust and engaging learning experience. We recommend combining the use of a traditional learning management system with interactive content developing software and a classroom response system to help fully engage students in the learning process both inside of class and at home. Educational technology are tools that are typically used to improve student-learning outcomes. However, emphasis must be made on truly 78 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

incorporating these tools within the learning process and throughout each stage of the study cycle. While the suggested study cycle embedded with educational technology does not encompass all plausible options for the vast educational tools currently available, it does offer a sound, feasible strategy for educators to consider when adopting a more technology-based learning environment centered on the learner and facilitated by the instructor. While the study was not exhaustive, preliminary evidence suggests that in-class engagement and student interest heightens when technology drives delivering the course material in an interactive rather than passive manner. Student perceptions of the use of the various technological tools varies, but there is a definite skew towards being more helpful rather than less helpful. Part of the acceptance by the students of the different delivery techniques (such as infusing more technological tools and less traditional lecture) is helping to ensure that students understand their role in the learning process. This includes familiarizing students with the study cycle in conjunction with accessing the various technological tools both in and out of class; it seems that students may have a higher disposition towards continuing to use the tools outside of the classroom.

References 1.

2.

3.

4.

5.

6. 7.

8.

Abdullah, K.; Clark, T.; Nasereddin, M. Using Kolb’s experiential learning cycle to improve student learning in virtual computer laboratories. Comp. Educ. 2014, 72, 11–22, DOI: 10.1016/j.compedu.2013.10.013. Pinheiro, M.; Simões, D. Constructing knowledge: An experience of active and collaborative learning in ICT classrooms. Proc. Soc. Behav. Sci. 2012, 64, 392–401, DOI: 10.1016/j.sbspro.2012.11.046. Piyayodilokchai, H.; Panjaburee, P.; Laosinchai, P.; Ketpichainarong, P. A 5E learning cycle approach-based, multimedia-supplemented instructional unit for Structured Query Language. Educ. Technol. Soc. 2013, 16, 146–159. Beatty, I.; Gerace, W. Technology-enhanced formative assessment: A research-based pedagogy for teaching science with classroom response technology. J. Sci. Educ. Technol. 2009, 18, 146–162, DOI: 10.1007/s10956-008-9140-4. Kolb, A. Y.; Kolb, D. A. Experiential learning theory. In Encyclopedia of the Sciences of Learning; Springer: New York, 2012; pp 1215−1219, DOI: 10.1007/978-1-4419-1428-6_227 Kolb, D. A. Experiential Learning: Experience as the Source of Learning and Development, 2nd ed.; Pearson Ed., Inc.: Upper Saddle River, NJ, 2015. Bergmann, J.; Sams, A. Flip your Classroom: Reach Every Student in Every Class Every Day; Intern. Soc. Technol. Educ.: Alexandria, VA, 2012, DOI: 10.1111/teth.12165 Beichner, R. J.; Saul, J. M.; Abbot, D. S.; Morse, J. J.; Deardorff, D. L.; Allain, R. J.; Risley, J. S. Student centered activities for large enrollment undergraduate programs (SCALE-UP) project. In Research-Based Reform 79

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

9.

10.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

11.

12.

13.

14.

15.

16.

17. 18.

19.

20.

21.

of University Physics; Reddish, E., Clooney, P., Eds.; American Association of Physics Teachers: College Park, MD, 2007; pp 1−42. Davies, R. S.; Dean, D. L.; Ball, N. Flipping the classroom and instructional technology integration in a college-level information systems spreadsheet course. Educ. Technol. Res. Dev. 2013, 61, 563–580, DOI: 10.1007/s11423013-9305-6. Baepler, P.; Walker, J. D. Active learning classrooms and educational alliances: changing relationships to improve learning. New Direc. Teach. Learn. 2014, 2014, 27–40, DOI: 10.1002/tl.20083. Stage, F.; Kinzie, J. Reform in undergraduate science, technology, engineering, and mathematics: The classroom context. J. Gen. Educ. 2009, 58, 85–105, DOI: 10.1353/jge.0.0038. Ertmer, P. A.; Ottenbreit-Leftwich, A. T.; Sadik, O.; Sendurur, E.; Sendurur, P. Teacher beliefs and technology integration practices: a critical relationship. Comp. Educ. 2012, 59, 423–435, DOI: 10.1016/j.compedu.2012.02.001. Drent, M.; Meelissen, M. Which factors obstruct or stimulate teacher educators to use ICT innovatively? Comp. Educ. 2008, 51, 187–199, DOI: 10.1016/j.compedu.2007.05.001. Pelgrum, W. J. Obstacles to the integration of ICT in education: results from a worldwide educational assessment. Comp. Educ. 2001, 37, 163–178, DOI: 10.1016/s0360-1315(01)00045-8. Shieh, R. S. The impact of technology-enabled active learning (TEAL) implementation on student learning and teachers’ teaching in a high school context. Comp. Educ. 2012, 59, 206–214, DOI: 10.1016/ j.compedu.2012.03.024. Committee on Risk-Based Approaches for Securing the DOE Nuclear Weapons Complex, a. N. R. C. N. N. Systemic Change: Barriers and Opportunities. In Promising Practices in Undergraduate Science Technology Engineering and Mathematics Education Summary of Two Workshops; National Academies Press: Washington, DC, 2011; pp 60−68, DOI: 10.17226/13099 Merç, A. Using technology in the classroom: A study with Turkish preservice EFL teachers. Turk. Onl. J. Educ. Technol. 2015, 14, 229–240. Baek, Y.; Jung, J. K. B. What makes teachers use technology in the classroom? Exploring the factors affecting facilitation of technology with a Korean sample. Comp. Technol. 2008, 50, 224–234, DOI: 10.1016/j.compedu.2006.05.002. Kim, C.; Kim, M. K.; Lee, C.; Spector, J. M.; DeMeester, K. Teacher beliefs and technology integration. Teach. Teach Educ. 2012, 29, 76–85, DOI: 10.1016/j.tate.2012.08.005. Harewood, E. Static Instruction Is Just That – Static, 2012. http:// evolllution.com/opinions/static-instruction-is-just-that-static/ (accessed March 28, 2016). Adams, C. PowerPoint, habits of mind, and classroom culture. J. Curric. Stud. 2006, 38, 398–411, DOI: 10.1080/00220270600579141. 80

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch004

22. Tuovinen, J. E. Multimedia distance education interactions. Educ. Media Intern. 2000, 37, 16–24, DOI: 10.1080/095239800361473. 23. Zimmer, T. Rethinking Higher Ed: A Case for Adaptive Learning, 2014. Forbes. http://www.forbes.com/sites/ccap/2014/10/22/rethinking-higher-eda-case-for-adaptive-learning/" \l "2418d5386293 (accessed July 14, 2016). 24. Levin, B. Putting students at the centre in education reform. J. Educ. Change 2000, 1, 155–172, DOI: 10.1023/A:1010024225888. 25. Kramer, S.; Kai, J.; Merline, F. J. A lesson for the common core standards era from the NCTM standards era: The importance of considering schoollevel buy in when implementing and evaluating standards based instructional materials. Large Scale Studies Math. Educ. 2015, 17–44, DOI: 10.1007/9783-319-07716-1_2. 26. Henderson, C.; Beach, A.; Finkelstein, N. Facilitating change in undergraduate STEM instructional practices: An analytic review of the literature. J. Res. Sci. Teach. 2011, 48, 952–984, DOI: 10.1002/tea.20439. 27. Haydn, T.; Barton, R. ‘First do no harm’: Factors influencing teachers’ ability and willingness to use ICT in their subject teaching. Comp. Educ. 2008, 51, 439–447, DOI: 10.1016/j.compedu.2007.06.00. 28. Heffernan, T.; Morrison, M.; Basu, P.; Sweeney, A. Cultural differences, learning styles and transnational education. J. Higher Educ. Pol. Manag. 2010, 32, 27–39, DOI: 10.1080/13600800903440535. 29. Smith, M.; Darfler, A. An exploration of teachers’ efforts to understand identity work and its relevance to science instruction. J. Sci. Teach. Educ. 2012March, 23, 347–365, DOI: 10.1007/s10972-012-9281-4. 30. North Carolina Agricultural and Technical State University. http:// www.ncat.edu/about/index.html (accessed July 11, 2016). 31. U.S. News and World Reports Best Colleges. http:// colleges.usnews.rankingsandreviews.com/best-colleges (accessed July 2016). 32. North Carolina Agricultural and Technical State University, Department of Chemistry Homepage. http://www.ncat.edu/cost/departments/chem/ (accessed July 2016). 33. North Carolina A&T State University Fact Book. ir.ncat.edu (accessed July 5, 2016). 34. Diverse Issues in Higher Education. http://diverseeducation.com/article/ 32493/ (accessed July 11, 2016). 35. Christ, F. L. Seven Steps to Better Management of Your Study Time; H & H Publishing: Clearwater, FL, 1997; pp 1−8. 36. McGuire, S. Teach Students How to Learn: Metacognition Is the Key, 2012. Louisiana State University, Center for Academic Excellence. https:// www.alabamacca.org/pdfs/2-ACCA-Birmingham-2012-Metacognition.pdf (accessed July 20, 2016). 37. Qualtrics. http://www.qualtrics.com (accessed December 15, 2015).

81 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

Chapter 5

Evaluating the Use of LearnSmart and Connect in Introductory General Chemistry Classes: The Pros and Cons of an Online Teaching and Learning System Rashmi Venkateswaran* *E-mail:

[email protected]

When asked if they like chemistry, many first-year university students often groan and say it is the hardest course they take. While part of this assessment at the University of Ottawa is based on the fact the course load is heavy due to a lecture component, a lab component and many tutorial sessions, another major reason for this statement comes from the fact that students often have difficulty in integrating the conceptual and problem-based aspects of the course. Chemistry requires students to understand ideas that range from microscopic to macroscopic, to read large bodies of text and understand how the text can be converted into chemical equations or visual images, and then to integrate all these concepts and apply them to the solution of mathematical problems. It takes a great deal of maturity, introspection and time to develop these skills and students in first year often simply do not have the time required to gain these skills given that chemistry is only one of the many courses that they take. Many textbook publishing companies have begun pairing their textbooks with online homework programs with a view to helping students practice and apply the knowledge they are learning from the textbook in an interactive manner. McGraw-Hill Education has taken steps to help students acquire the necessary skills using online technology that builds a basis for their knowledge and then allows them to apply that same knowledge. McGraw-Hill Education Connect®, the online teaching and learning program © 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

that is available with McGraw-Hill Education textbooks, comes paired with SmartBook® with LearnSmart® in many cases. As the instructors of general chemistry at University of Ottawa chose the McGraw-Hill Education text, this chapter will present a preliminary evaluation, as well as some of the advantages and disadvantages of the SmartBook vis a vis a standard or e-book. It will also look at whether the integration of SmartBook helps students to learn to determine what is important in a text, how to reinforce that knowledge and gain an understanding of the underlying concepts, and how to apply that knowledge to the solution of problems. Did pairing the conceptual knowledge gained by SmartBook with concrete problems provide a holistic approach to the study of chemistry and what were the advantages and disadvantages, given some of the student limitations mentioned above? While any homework takes time, whether paper or online, this chapter will present some information about SmartBook and an informal evaluation of whether use of this system was able to provide a successful learning outcome for students.

Introduction One of the more challenging courses taken by first-year science students is the Introductory General Chemistry course (1). Evaluations obtained from students who have taken this course at University of Ottawa over several years have consistently demonstrated that they perceive the workload to be very heavy. It is true that the course contains several components including at least two lectures, a lab, course-related tutorial or problem sessions, and in some cases lab-related tutorials for assistance in completion of lab reports. In order to succeed in the course, students must learn the material, study for midterms and exams, complete homework assignments, perform experiments, and submit lab reports. Informal comments and discussions with students, in addition to responses to a direct question regarding workload found on student evaluations, indicate that students consider chemistry to have a heavier workload than most of their other courses. An actual comparison of science courses however seems to indicate that a similar workload is expected in other first year science courses. It appears that students may perceive incorrectly that introductory general chemistry has a greater workload. Anecdotal evidence also indicates that students have difficulty in relating the chemical equations and scenarios explained in lecture or class to the experiments they perform in the lab, suggesting that there is a disconnect between the microscale (chemical nomenclature, chemical equations, atomic/molecular depictions) and the macroscale (real-world connections, experimental observations, lecture demonstrations) (2–6). A considerable effort is expended by most general chemistry instructors to offer a variety of tools to assist students in overcoming these learning challenges. Some instructors focus on helping students to understand the basic concepts or 84 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

to improve their conceptual understanding in chemistry (7–10). Other instructors work on correcting misconceptions in order to help students learn chemistry (11–13). Many instructors struggle with the need to address concepts and yet ensure students are capable of solving problems (14, 15). However, financial constraints and ease of use invite many instructors to turn to online resources (16, 17), some created by the instructor (18, 19) and some provided by textbook publishers (20–22).

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

Online Resources Most textbooks today come with a great deal of electronic support, much of which can be found online. This support generally consists of an electronic version of the textbook, online homework programs, access to videos and/or animations and various other accessories, such as flash cards, concept inventories and so on. The dwindling physical resources that most instructors have available coupled with larger class sizes makes it increasingly difficult to assign homework to students that can be corrected by either a teaching assistant or the instructor. Online homework is a convenient alternative that allows instructors to assign homework in a timely manner without the necessity of having an assistant take the time to make corrections. Further, it allows students to practice chemistry and get timely feedback. In order to have a system in which the correction can be completed without the intervention of an instructor or teaching assistant (which is the feature that makes online homework most attractive), the majority of the questions are of the fill-in-the-blank or multiple choice category. While such questions are not ideal from a teaching and learning point of view, nonetheless, students have the opportunity to practice learning terminology and conceptual chemistry as well as the chance to apply their understanding in the form of problems. Often, students who do not see their response anywhere in the answer list realize that they must have made an error somewhere. Extremely good multiple choice questions have common student errors as false responses, thus requiring students to actually understand the underlying theory in order to obtain the correct response. Students, however, have developed a variety of methods to deal with multiple choice questions. They are often capable of using logic to eliminate certain responses. Other times, they will guess with no real understanding of why they are choosing a certain response. Possibly the most pernicious problem of any online homework is that some students share answers on social media; if students are only interested in receiving points, this becomes an issue very difficult to resolve. Some of the commonly used online homework systems include Sapling, Mastering Chemistry used by Pearson, OWL by Cengage Learning and used with Nelson Education texts, WileyPLUS used by Wiley, and Connect used by McGraw-Hill. Mastering Chemistry, which has been used previously at University of Ottawa, contains tutorial-type questions that have a feedback system allowing students who are experiencing difficulty to request a hint. The hint then guides the student towards the correct response by helping the student to arrive at one of the values necessary to obtain the correct response. Sometimes there are hints for each step of a multi-step calculation. The other systems named 85 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

above have varying approaches that provide similar support to students. All the publishers mentioned above provide an electronic version of the textbook (e-book) along with the online homework system. The textbook is still considered a primary resource, but most general chemistry textbooks are quite large and include more information than is customarily taught in any given course. Thus many students, when reading, find it difficult to distinguish what content is truly important and what content can be studied more leisurely. Often, students will read every word of a textbook, treating all parts of the text equally. This leads to an inability to judge what content is most likely to be tested, resulting in students finding that what they studied was not on the test. In meetings with students after tests or exams over a number of years, when asked why they felt the content on the test did not reflect what they studied, many students confessed that they read the text in its entirety without focusing on the topics specifically targeted in class.

What Is SmartBook? Most McGraw-Hill Education textbooks are supported by Connect and SmartBook. SmartBook, the adaptive online textbook, differs from e-books offered by other publishers in ways that can be highlighted by instructors. Some of these differences that have been used at the University of Ottawa to help students approach their chemistry learning in new ways will be explained with examples. The first way in which SmartBook differs is that it helps students to prioritize the information as they are reading the text. In any particular chapter, important information is highlighted and information that is provided for interest, background or context is lightly greyed, as shown in Figure 1 (23). This does not mean that the information in the greyed text is unimportant; it simply indicates to the student that this information can be referred to once the student has grasped the key concepts or if the student wishes to have additional information or context to the material they are reading. This is helpful as students often have difficulty in determining what information they must know as opposed to what information it would be nice for them to know. SmartBook helps them to identify the information they must know. The decision regarding what text is highlighted and what text is greyed out is not made by the textbook authors, but rather by subject experts who are also specialists in cognitive learning. Once students have read through a certain number of sections, they are then encouraged to put into practice what they have read. SmartBook integrates LearnSmart questions, offering a metacognitive assessment that allows them to determine whether they have correctly understood the terminology, the concepts and simple applications of the material they have read to this point.

86 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

Figure 1. Example of text as it appears in SmartBook. (Reproduced with permission from reference (23). Copyright 2013 McGraw-Hill Education.)

LearnSmart and Metacognition What sets LearnSmart questions apart is the metacognitive component. The LearnSmart questions were designed by a team of experts in adaptive learning technology who are also subject experts, and these questions are designed to guide student learning based on student response. Metacognition has been assigned a number of different definitions, but one of these definitions includes the ability to assess whether a particular response is correct or incorrect (24). The adaptive technology takes students responses and based on these responses designs a pathway of questions. The assignment comes with a metacognitive component that asks a chemistry question and then requires students to respond to the question “Do you know the answer” with one of the following choices: “I know it”, “Think so”, “Unsure”, or “No idea”, as shown in Figure 2 (23).

87 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

Figure 2. Example of a LearnSmart question showing the metacognitive component. (Reproduced with permission from reference (23). Copyright 2013 McGraw-Hill Education.) Subsequently, students are asked to enter their response to the chemistry question, as shown in Figure 3 (23).

Figure 3. Example of LearnSmart question. (Reproduced with permission from reference (23). Copyright 2013 McGraw-Hill Education.) 88 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

LearnSmart adapts the next question based on the students question response and their metacognitive response. Thus a student who answered “I know it” and gave the correct response to the first question, as shown in Figure 4 (23), would have a different second question than a student who responsed “Think so” and responded correctly as well.

Figure 4. Example of a correct response to a LearnSmart question. Note the “Challenge” button appears whenever an answer is submitted. Students can challenge the system when they believe an answer is correct but is marked as incorrect. (Reproduced with permission from reference (23). Copyright 2013 McGraw-Hill Education.) If a student consistently responds metacognitively saying “I know it” and gets the incorrect response to the question, LearnSmart will suggest to the student that it would be a good idea to go back and read the text material to reinforce their understanding, as shown in Figure 5 (23). While a particular assignment may contain 20 questions, a student can answer anywhere from 20 to 40 questions depending on their responses. The assignment is only complete when the student has actually responded to the original 20 questions with a reasonable degree of certainty. If there are incorrect or very uncertain responses, LearnSmart questions will continue, but return to the areas of uncertainty or error, asking different questions until the student can respond with assurance. Further, SmartBook notes where the student displayed uncertainty and will reinforce these areas by selectively highlighting the text that will explain the student’s doubts. 89 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

Figure 5. Suggestion by SmartBook that a student should read more before testing. (Reproduced with permission from reference (23). Copyright 2013 McGraw-Hill Education.)

Students are also encouraged to “recharge” or to practice again at another time; at that time, the previous areas of weakness are tested again to reinforce that particular knowledge with the goal that the student has acquired sufficient confidence in and understanding of the subject matter to effectively retain the information. SmartBook accesses a data set consisting of tens of thousands of student data results and over five billion probes used to identify patterns for when students are likely to forget, as well as the frequency and sequence in which content needs to be reinforced for long term memory. The algorithm guides the appropriate probe types and frequency based on the data set and then optimizes based on individual student results. It is essentially a massive data set for probability guidance combined with student specific input. SmartBook, when regularly used with the practice assignments in LearnSmart, is designed to help students move information from short-term memory to long-term memory.

Connect McGraw-Hill Education Connect is the online teaching and learning program that is available with most McGraw-Hill textbooks. SmartBook can be accessed through Connect; in addition, Connect has assignable end-of-chapter static and algorithmic problems that students can complete, either as homework, practice or assessment. There are true/false, fill in the blank, and multiple choice type questions. There are also numeric problems that require that the student enter a numeric response, as shown in Figure 6 (23). 90 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

Figure 6. Example of an algorithmic problem in Connect. (Reproduced with permission from reference (23). Copyright 2013 McGraw-Hill Education.) An assignment can be constructed by using any one of the question types or by having a selection of question types. Many questions can be made algorithmic; in an algorithmic question, different parameters within a question are treated as variables. In effect, although a single question is assigned, if a student repeats the question, they will see a similar question with one or more of the quantities having a different value. This allows students to practice a problem without memorizing a rote solution and hopefully allows the students to see that problems that appear different on the surface are not that different in the ways in which their solutions can be approached. As with most programs, there are hints at certain points in the response, allowing students who are unable to solve the problem to get guidance in how to approach the solution to the question.

Preliminary Evaluation of SmartBook in a General Chemistry Class I used the online support provided by McGraw-Hill Education along with the textbook Chemistry: The Molecular Nature of Matter and Change, Canadian Edition, by Silberberg, Lavieri and Venkateswaran (23) to teach a group of approximately 85 students taking introductory general chemistry at the University 91 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

of Ottawa, in Ottawa, Canada. These students were in a section of the course intended mainly to support students who either had not taken the prerequisite chemistry course in high school, had taken the course and had not done well in it, or who were mature students returning to University or to a program in Science. The course was structured so as to provide them with three 1.5 hour classes each week. There was a laboratory component (3 hour biweekly) and a strongly recommended (but not mandatory) course tutorial and lab tutorial. Students were assigned a LearnSmart assignment for each chapter and were required to complete the assignment for the beginning of the class. They were also assigned a set of problems in Connect which were due by the date of the quiz for that chapter. Students who purchased the textbook package were given access to Connect with SmartBook automatically. Students had the option of purchasing access to the online components separately.

How Online Homework Components Were Weighted The LearnSmart assignments were assigned a weight of 5% and the Connect assignments a similar weight, overall giving a homework component of 10%. Students were informed during the first class that they could choose to opt out of doing the online homework, in which case the 10% weight would be transferred to their final exam. Of the 85 students, 5 students chose to opt out, 2 for financial reasons. The questions from the homework were not reviewed in class unless students explicitly requested that a question be explained. In general, students did not ask any questions regarding the LearnSmart homework. If students had any issues with the software, they were encouraged to contact McGraw-Hill Education support directly. The majority of issues were in regard to altering due dates since the course material was fluid and largely based on the students’ ability to learn the key concepts and principles.

Groupwork Students occasionally requested guidance or help with some of the more difficult problems from Connect (usually corresponding to one or two end-of-chapter questions in the more difficult chapters). These questions were then given as group work during class so that 2-4 students per group could work collaboratively to find a solution, with occasional assistance from other groups or myself if needed. Additionally, students worked in groups in class and in tutorials to solve more complex, integrated type problems from the end-of-chapter questions in the textbook, to write new problems or to identify areas of difficulty. With the exception of the introductory chapters (background to gas laws) , equilibrium (gas phase, acid/base and solubility) and atomic/molecular structure (the nature of the quantum atom, electron configurations and periodicity, Lewis structures and VSEPR) for which multiple chapters were included in a single test, each chapter was tested separately. 92 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

How Did Students Do? Although no statistical or formal research studies were carried out, students who actively participated in the online homework (either the LearnSmart fully, the Connect fully or both to at least 70%) did obtain better final grades in the course. Students who were already understanding the material tended to get higher marks on the homework component based on a comparison of the students quiz grades for the chapter with their homework grades. There was a significant portion (as much as 25%) of the class that had difficulty with understanding both the concepts and with applying the theory to solving problems. These students, who may well have benefited significantly from doing the online homework, were the ones who had the lowest percentages of completion of the homework exercises. Whether this was because they were overwhelmed with the workload of the course, or because they were unable to spend the time required to complete the exercises is uncertain. The students who benefited most were those in between, that is, neither the best nor the worst. These students completed an average of 60-100% of the homework assignments and there was a clear increase in their final exam grade as compared to their quiz grades. With regards to the metacognitive component of the LearnSmart questions, it may appear at first glance that the question is asking how confident the student is with the response, which may not appear in reality to be an aspect of metacognition. One of the broad definitions of metacognition is reflection on one’s own learning or understanding (25, 26). Ideally, for a student to respond to the question “Do you know the answer” in any way at all, the student would have had to think about whether they really do know the answer and in so doing, analyze why they think they do or do not know the answer. That process constitutes a metacognitive prompt that none of the other online homework programs contains. Whether students actually go through this process, however, or whether they simply randomly click on one of the four choices is something much more difficult to determine and no questions regarding this particular process were specifically asked of the students in the class. Student Feedback Many students offered anecdotal evidence of their experience with the online homework. The majority of students felt there was too much work required for the course and that the homework was just one more thing for them to do, offering comments along the lines of “overwhelming” or “so much homework” in response to the question “How could the course and/or the teaching be improved?”. Nevertheless, a large number (more than 50%) felt that the homework did help them to improve their understanding of the concepts, the language used, the chemical theory connecting the microscopic with the macroscopic, and the ability to apply this theory to solve problems (comments similar to “Learnsmart helped”, in response to the question “What did you like about the course and/or the teaching?”). Most notably, many students strongly expressed that SmartBook helped them to learn how to read the textbook. Previously, they would read every word on every 93 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

page, assigning equal importance to all the text they read. SmartBook helped them to determine which parts of the text were more relevant and guided their focus. Students frequently expressed surprise at the fact that content that they had thought was important turned out not to be important and conversely, content that they did not think they needed to know was highlighted as being an area of focus. Once they started using SmartBook and became accustomed to it, students who used to say that the content of a quiz completely took them by surprise and that even though they studied, they were not prepared, began saying that the test was reasonable but long. In other words, although they were having trouble completing the material within the specified time, they were expecting to see the content that was on the test. This was one of the most positive outcomes of using SmartBook. Students who used SmartBook felt that their understanding of the terminology improved. They were able to differentiate between words that they had previously felt meant the same thing (for example, heat and temperature, or frequency and wavelength). They were also able to define the terminology more clearly and with less ambiguity. They were more comfortable with the units associated with quantities. One of the features of LearnSmart questions that they appreciated was the “Challenge” button, that appears after an answer has been submitted, as shown previously in Figure 4. If they entered a response that they felt was correct but they were told that their response was incorrect, they had the option to challenge the response. The challenge was sent directly to those who had written the questions and a response was received within 48 hours. If several students challenged the same question, the question was modified and instructors were notified. Students felt empowered by the opportunity to challenge what they felt was an incorrect grading and appreciated being told that their response was correct or being informed as to why their response was incorrect. They also found it interesting that they did not necessarily get the same questions as their friends even if they did the assignment at the same time, since the branching of the questions depended on their responses as well as their confidence level. It did take a while for students to understand that an assignment with 20 questions sometimes took a very long time to complete if they were providing incorrect responses to multiple questions. Many students found that frustrating at first, but soon realized that reading the text before doing the assignments made it easier, faster and more interesting to do the assignment. Room for Improvement Those students who found the experience frustrating spent much more time on the LearnSmart assignment (as indicated by reports showing how much time was spent by individual students and also which questions were incorrect). These students had difficulty with the concepts and terminology from the beginning of the course (based on quiz grades), and felt that the time it was taking them to complete the LearnSmart exercise was not really helping them to learn (based on end-of-semester evaluations). This was definitely one of the negative aspects of LearnSmart. However, as with any homework system, the time spent on doing the homework does have a direct impact on student understanding (27).Overall, 94 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

students stated that they felt that LearnSmart was useful and helped them to improve their grade. Connect received more mixed reviews than either LearnSmart or SmartBook. The mixed review was likely due to the selection of problems on the assignments. Problems of higher order thinking and of an integrated nature were selected for the assignments so as to provide an opportunity for students to practice problems of the type that they would typically see on a test or exam. However, students reported that they felt unprepared to approach such problems as the only background they had was with the concepts and ideas taught through LearnSmart and the example type problems done when needed in class. Students felt that they were spending too much time trying to find a way to approach the problems and rarely actually got to solve them. Student Suggestions for the Future Several students suggested that a better approach would be to use Connect to teach the individual steps required to solve a problem (for example, how to convert mass to amount, how to use Hess’ law to find enthalpy of reaction, or how to use an ICE table) and then to have the students work in groups to solve a more complex, integrated problem using these individual steps. This modification would only necessitate a different selection of questions for the assignment and this approach will be tried in the next iteration of the course. Fewer questions in each assignment was also a suggestion that was endorsed by many of the students. With the exception of these two issues, students found the homework forced them to stay current with the material and did lead them to ask more questions in class, which had a net positive effect. When informally asked whether they felt more engaged in class, several students commented that they found that having to complete the LearnSmart exercises before class meant they had to read the chapter in advance. They could then pose questions in class regarding content they did not understand, or content that interested them, which kept them engaged in the course. Overall, with this particular group of students, the combination of Connect and SmartBook had a very positive effect on student learning, on quiz grades, on the final exam grade and overall on final course grades. McGraw-Hill Education Canada Case Studies In general, in other courses where Connect and SmartBook are used, the outcome has been similarly positive (28, 29). Results obtained from a survey conducted by McGraw-Hill Education Canada in 2016 among 2100 Canadian college and university students indicated that 81% of the respondents felt that SmartBook helped to make the course more interesting and engaging. When compared with e-books, more than 83% of the students consistently indicated that they found the SmartBook helped them retain information for longer periods of time, kept them engaged, helped them use their time more effectively, helped them prepare for class and helped them improve their grades. Over 75% of the students felt similarly about SmartBook when compared to a standard print textbook. 95 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Some of the written comments provided similar sentiments about SmartBook, and repeatedly comments expressing the idea “fantastic!”, “gave me confidence”, “interesting and useful”, “loved it”, and “felt very prepared”, were offered by students. Many of the negative comments associated with SmartBook related to the cost of the product, such as “Overpriced”. Most interestingly, students were asked if they would choose to purchase Connect with SmartBook for a future course even if their instructor chose not to assign grades for it and over 62% of students said that they would.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

Instructor Point of View While the utility of SmartBook to students has been discussed to this point, it is essential to examine SmartBook from the point of view of instructors as well. It is possible to get statistics on student responses to the LearnSmart questions that include the metacognitive component. The area of least concern is those students who think they know the correct answer and who also submit the correct response. The area of greatest concern is those students who think they know the correct answer and yet submit an incorrect response. These students are usually overconfident and mistakenly think that their understanding is complete; this group of students often encounters the greatest difficulty on tests and exams. SmartBook allows such students to be identified in advance and it is possible to address learning gaps. The students also have access to these reports and so they can use them to understand and address their own knowledge deficiencies. While it is less troublesome, the other group that merits concern is the group of students who indicate that they are unsure or that they do not know the correct response but still respond correctly. Generally, such students are lacking in confidence and while they have the knowledge, they are hesitant to apply it and thus may perform poorly on tests and exams. Once identified, it is possible to meet with these students and encourage them to be more confident in their knowledge. One key benefit of SmartBook lies in its ability to assist instructors in determining where students are having difficulty and in guiding them in the most helpful and appropriate way towards a better understanding of the course content so that the greatest number of students can succeed in the course. There are many other reports that can be generated that give specific information about particular students, specific information for all students and average or general values for all students. Once a particular session of a course is complete, these reports can be used to help instructors assess what teaching strategies were useful to students and which techniques may need to be reassessed to determine their usefulness. The major disadvantage to SmartBook is having to deal with the frustration of students who resent having to spend more time than the assignment is supposed to take. When an instructor sets the assignment, there is an indication of the number of questions and how long it should take to complete the assignment. However, if students have difficulty with the questions or if their metacognitive response and question response are not aligned, the adaptive learning software creates additional questions to reinforce learning. As a result, in an assignment with 20 questions, a student may end up answering 40 questions and taking significantly more time 96 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

to complete the assignment than expected. This frustration and resentment on the part of the student is often conveyed to the instructor. One potential way to address this issue is to openly inform students of the purpose and value of the adaptive learning system, and the reason assignments may take longer than expected. It does not resolve the issue, but may help students to deal with the frustration more purposefully.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

Conclusion While there are a number of tools, both physical and online as well as multiple in-class strategies used to help them, students continue to struggle with the introductory general chemistry course. The McGraw-Hill Education online support in the form of Connect and SmartBook provides a powerful advantage to both students and instructors. Targeted, adaptive techniques that include metacognitive prompts allow students to read with focus, practice what they have read with understanding and apply what they have learned with confidence, leading to deeper learning over a longer period of time. Instructors are able to see where students are having difficulty, either in actual learning or their own perception of their learning, and apply constructive support to correct misconceptions, address learning gaps and bolster or temper confidence, depending on the individual student’s needs. The online tools, paired with in-class activities such as think-pair-share exercises or active group work provides a holistic approach to helping students grasp the essential concepts and terminology in the course and to then apply these ideas to solve more complex, real world, or open-ended problems. Student feedback on how best to apply these tools must be taken into consideration to provide the best learning experience possible. Building a pathway of learning using stepwise support is a logical plan and one that agrees with student comments in evaluations. An explanation of how this system differs from other homework systems and the value of thinking metacognitively at the beginning of the course would also help students understand what the purpose of the homework is. Hopefully, this will also encourage students to think about their answer to the metacognitive part of the response (which encourages metacognition on multiple levels) and help them to learn more efficiently. Based on student feedback and instructor experience with SmartBook, future plans of study could include a comparison of the ability of SmartBook to help students learn as compared to other publishers’ e-books, a study of whether SmartBook makes the students reliant on what text to read or actually helps them to learn how to read a text independently, and whether students actually use a metacognitive process to answer the question “Do you know the answer”. With SmartBook providing strong support for the underlying ideas and chemical concepts, and Connect used to give students the opportunity to see the different ways in which these ideas and concepts can be applied to solve rudimentary chemical problems, instructors can use valuable in-class time to encourage students to work together to solve problems that require students to make connections between different ideas in chemistry, that need students to 97 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

think beyond the normal parameters to which they are accustomed and to work collegially and combine their individual strengths to arrive at a solution.

References 1. 2.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

3. 4. 5.

6.

7. 8. 9.

10.

11. 12.

13.

14. 15.

16. 17.

Laidler, K. J. Too much to know. J. Chem. Educ. 1974, 51, 696–700, DOI: 10.1021/ed051p696. Carter, C. S.; Brickhouse, N. W. What makes chemistry difficult? Alternate perceptions. J. Chem. Educ. 1989, 66, 223–225, DOI: 10.1021/ed066p223. Herron, J. D. Using Research in chemical education to improve my teaching. J. Chem. Educ. 1984, 61, 850–854, DOI: 10.1021/ed061p850. Bodner, G. M. Constructivism: A theory of knowledge. J. Chem. Educ. 1989, 63, 873–878, DOI: 10.1021/ed063p873. Champagne, A. B.; Klopfer, L. E.; Gunstone, R. F. Cognitive research and the design of science instruction. Educational Psychologist 1982, 17, 31–53, DOI: 10.1080/00461528209529242. Chiu, M. H.; Chou, C. C.; Liu, C. J. Dynamic processes of conceptual change; Analysis of constructing mental models of chemical equilibrium. J. Res. Sci. Teach. 2002, 39, 688–712, DOI: 10.1002/tea.10041. Bergquist, W.; Heikkinen, H. Student ideas regarding chemical equilibrium. J. Chem. Educ. 1990, 67, 1000–1003, DOI: 10.1021/ed067p1000. Nakhleh, M. B. Student’s models of matter in the context of acid-base chemistry. J. Chem. Educ. 1994, 71, 495–499, DOI: 10.1021/ed071p495. Krishnan, S. R.; Howe, A. C. The mole concept: Developing an instrument to assess conceptual understanding. J. Chem. Educ. 1994, 71, 653–655, DOI: 10.1021/ed071p653. Furio, C.; Azcona, R.; Guisasola, J.; Ratcliffe, M. Difficulties in teaching the concept of amount of substance and mole. Int. J. Sci. Educ. 2000, 22, 1285–1304, DOI: 10.1080/095006900750036262. Hackling, M. W.; Garnett, P. J. Misconceptions of chemical equilibrium. Eur. J. Sci. Educ. 1985, 7, 205–214, DOI: 10.1080/0140528850070211. Nakhleh, M. B. Why some students don’t learn chemistry: Chemical misconceptions. J. Chem. Educ. 1992, 69, 191–196, DOI: 10.1021/ ed069p191. Gorodetsky, M.; Gussarsky, E. Misconceptualization of the chemical equilibrium concept as revealed by different evaluation methods. Eur. J. Sci. Educ. 1986, 8, 427–441, DOI: 10.1080/0140528860080409. Nakhleh, M. B.; Mitchell, R. C. Concept learning versus problem solving: There is a difference. J. Chem. Educ. 1993, 70, 190–192. Nakhleh, M. B. Are our students conceptual thinkers or algorithmic problem solvers? Identifying conceptual students in general chemistry. J. Chem. Educ. 1993, 70, 52–55, DOI: 10.1021/ed070p190. Diener, L. Selected online resources for teaching about alternative energy. J. Chem Educ. 2012, 89, 950–952, DOI: 10.1021/ed200068y. Kirchhoff, M. Online resources for teacher and students from the American Chemical Society. J. Chem. Educ. 2009, 86, 127, DOI: 10.1021/ed086p127. 98

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch005

18. Donovan, W. J.; Nakhleh, M. B. Students’ use of web-based tutorial materials and their understanding of chemistry concepts. J. Chem. Educ. 2001, 78, 975–980, DOI: 10.1021/ed078p975. 19. Shields, S. P.; Hogrebe, M. C.; Spees, W. M.; Handlin, L. B.; Noelken, G. P.; Riley, J. M.; Frey, R. F. A transition program for underprepared students in general chemistry: Diagnosis, implementation and evaluation. J. Chem. Educ. 2012, 89, 995–1000, DOI: 10.1021/ed100410j. 20. Eichler, J. F.; Peeples, J. Online homework put to the test: A report on the impact of two online learning systems on student performance in general chemistry. J. Chem. Educ. 2013, 90, 1137–1143, DOI: 10.1021/ed3006264. 21. Richards-Babb, M.; Curtis, R.; Georgieva, Z.; Penn, J. H. Student perceptions of online homework use for formative assessment of learning in organic chemistry. J. Chem. Educ. 2015, 92, 1813–1819, DOI: 10.1021/acs.jchemed.5b00294. 22. Evans, J. A. OWL (Online Web-Based Learning) (published by Cengage-Brooks/Cole). J. Chem. Educ. 2009, 86, 695–696, DOI: 10.1021/ed086p695. 23. Silberberg, M. S.; Lavieri, S.; Venkateswaran, R. Chemistry : The Molecular Nature of Matter and Change, 1st CE; McGraw-Hill Education: Canada, 2013. 24. Dunning, D; Johnson, K; Ehrlinger, J.; Kruger, J. Why people fail to recognize their own incompetence. Current Directions in Psychological Science 2003, 12, 83–87, DOI: 10.1111/1467-8721.01235. 25. Rickey, D.; Stacy, A. M. The role of metacognition in learning chemistry. J. Chem. Educ. 2000, 77, 915–920, DOI: 10.1021/ed077p915. 26. Cooper, M. M.; Sandi-Urena, S. Design and validation of an instrument to assess metacognitive skillfulness in chemistry problem solving. J. Chem. Educ. 2009, 86, 240–245, DOI: 10.1021/ed086p240. 27. Leinhardt, G.; Cuadros, J.; Yaron, D. “One firm spot”: The role of homework as lever in acquiring conceptual and performance competence in college chemistry. J. Chem. Educ. 2007, 84, 1047–1052, DOI: 10.1021/ed084p1047. 28. Welch D., Franklin University, Columbus, Ohio, Case Study, McGraw-Hill, 2010. 29. Independent case study of over 700 students studying Anatomy and Physiology I at six distinct institutions, McGraw-Hill, 2012.

99 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Chapter 6

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

Faculty Goals, Inquiry, and Meaningful Learning in the Undergraduate Chemistry Laboratory Stacey Lowery Bretz,* Kelli Rush Galloway, Joanna Orzel, and Elizabeth Gross Department of Chemistry & Biochemistry, Miami University, Oxford, Ohio 45056, United States *E-mail: [email protected]

Faculty goals for learning in the undergraduate General Chemistry and Organic Chemistry laboratory were measured. The experiments they selected for the laboratory courses were characterized with regard to inquiry. Students in these courses were asked to report their expectations and experiences with regard to meaningful learning. Analysis of these three data sets showed that faculty goals do not always align with their experiments and that there is little connection between faculty goals and students’ learning.

Introduction From Ira Remsen’s wonder at his remarkable observations upon dropping a copper penny into nitric acid, to Oliver Sacks’ tales of his childhood explorations in Uncle Tungsten (1), chemists have long understood the importance of hands-on experimentation in the laboratory to learning chemistry. Several reviews document the chronology of the role of laboratory in chemistry education from the early 19th century through the next two centuries (2–6). Given the nearly ubiquitous existence of the teaching laboratory in undergraduate chemistry courses, it is surprising how little evidence exists to support the widely held view that laboratory courses are essential:

© 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

“Laboratories are one of the characteristic features of education in the sciences…rare to find any science course…without a substantial component of laboratory activity. However, very little justification is normally given… assumed to be necessary and important (7).”

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

“…research has failed to show simple relationships between experiences in the laboratory and student learning (5).” “Duplicating what we chemists do in our laboratories (or what chemists of earlier generations used to do) does not enhance students’ understanding of chemistry’s centrality, but makes chemistry an irrelevance. Laboratory classes do not help students to understand how chemical principles affect their universe...The most important issue in the context of laboratory classes is whether there needs to be a laboratory program at all (8).” In addition to sparse evidence regarding its effectiveness, the costs of laboratory instruction must also be considered: reagents that are ordered and consumed each year for hundreds of thousands of students, disposal and treatment of waste, as well as stipends and tuition for graduate student teaching assistants. The question must be asked: do laboratory courses in chemistry warrant their costs? Surely an argument could be constructed that labs are worth the significant financial infrastructure because carefully articulated goals for learning in the chemistry laboratory would lead to purposefully chosen experiments in the laboratory curriculum, culminating in meaningful learning and students’ experiences (Figure 1). Framing such an argument requires data, of course. This paper reports the findings of a research study designed to investigate alignment between faculty goals for laboratory learning and the experiments selected for students to carry out in undergraduate General Chemistry and Organic Chemistry laboratory courses.

Figure 1. Ideally, students’ expectations for learning in the undergraduate chemistry laboratory and their experiences would be influenced by the laboratory curriculum which was constructed in order to align with faculty goals. 102 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

Methods Two studies were designed to investigate the alignment (or lack thereof) across faculty goals, laboratory curriculum, and student expectations and experiences (Figure 2). Study 1 analyzed faculty goals for the undergraduate laboratory using a previously published instrument by Bruck and Towns (9) and investigated whether these goals corresponded to the experiments chosen for the laboratory curriculum which was analyzed using a previously published rubric to characterize the level of inquiry in the experiments (10, 11). Study 2 examined these same faculty goals by comparing them to their students’ responses on a previously published instrument that measures meaningful learning in the undergraduate chemistry teaching laboratory (12). The research protocol was approved by the Institutional Review Board and all respondents provided informed consent.

Figure 2. The trio of instruments used to collect data and analyze alignment across faculty goals, laboratory experiments, and students’ expectations and experiences in the undergraduate chemistry laboratory. Research Questions Study 1: How are faculty goals for General Chemistry and Organic Chemistry laboratory aligned with their selected laboratory experiments as characterized by their level of inquiry? Study 2: How are faculty goals for General Chemistry and Organic Chemistry laboratory aligned with students’ expectations and experiences? Sample Faculty whose students had completed the Meaningful Learning in the Laboratory Instrument (13) were invited to also complete the Faculty Goals Survey (9). A total of 34 faculty responded (Table 1). These faculty provided copies of the experiments for their General Chemistry (GC) or Organic Chemistry (OC) laboratory courses. A total of 289 experiments were analyzed across four courses: 103 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

• • • •

General Chemistry I (N=145 experiments) General Chemistry II (N=61 experiments) Organic Chemistry I (N=73 experiments) Organic Chemistry II (N=10 experiments)

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

Table 1. Chemistry faculty who responded to the Faculty Goals Survey Institution Type

General Chemistry (GC)

Organic Chemistry (OC)

Total

Community College

1

0

1

Liberal Arts

1

2

3

Comprehensive

10

3

13

Research University

10

7

17

Total

22

12

34

Instruments Faculty Goals Survey The Faculty Goals Survey (FGS) was developed by Bruck and Towns (9) in order to quantitatively measure the prevalence of chemistry faculty’s goals for learning in the undergraduate chemistry laboratory that had previously been reported in in-depth qualitative studies (14, 15). The FGS consists of 29 items across 7 factors (research experience, group work, error analysis, connections between laboratory and lecture, transferable skills (both lab specific and not), and laboratory writing) to which Faculty respond to FGS items using a Likert scale of 1 (strongly disagree) to 6 (strongly agree). The FGS was administered using the online survey tool Qualtrics. Data were analyzed using the statistics package SPSS, and descriptive statistics, including histograms, were calculated for each FGS item by course. Each FGS item was coded a priori as cognitive, affective, or psychomotor using Novak’s Meaningful Learning framework (16, 17). Interrater agreement was calculated among three researchers until consensus was reached regarding the cognitive, affective, and psychomotor codes. The same FGS data set was drawn upon for both Study 1 and Study 2.

Inquiry Rubric The inquiry rubric characterizes 6 dimensions (see Table 2) of a laboratory experiment and characterizes them as either provided to students or not. The sum of these six elements in a laboratory experiment provides an indication of the “degrees of freedom” that a student has in making choices about what problem 104 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

to investigate, data to collect, data to analyze and against what theory, as well as how to communicate results and what conclusions can be drawn. As the degrees of freedom increase, the experiment can be considered to offer more opportunities for inquiry, ranging from confirmation (all procedural details have been selected by faculty and provided to students) to authentic inquiry (no details have been provided regarding the theoretical, experimental, or analytical choices to be made).

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

Table 2. Inquiry rubric to evaluate undergraduate chemistry laboratory experiments Level 0 Confirmation

Level ½ Structured Inquiry

Level 1 Guided Inquiry

Level 2 Open Inquiry

Level 3 Authentic Inquiry

Problem/ Question

Provided

Provided

Provided

Provided

Not Provided

Theory/ Background

Provided

Provided

Provided

Provided

Not Provided

Procedures/ Design

Provided

Provided

Provided

Not Provided

Not Provided

Results Analysis

Provided

Provided

Not Provided

Not Provided

Not Provided

Results Communication

Provided

Not Provided

Not Provided

Not Provided

Not Provided

Conclusions

Provided

Not Provided

Not Provided

Not Provided

Not Provided

*

Data sourced from Bruck, L.B.; Bretz, S.L.; Towns, M.H. Characterizing the level of inquiry in the undergraduate laboratory. J. Coll. Sci. Teach., 2008, 37, 52-58.

In Study 1, each of the 289 experiments was evaluated using the inquiry rubric. Inter-rater reliability was calculated between two researchers using Cohen’s Kappa, which equaled 0.875 (18).

Meaningful Learning in the Laboratory Instrument (MLLI) The Meaningful Learning in the Laboratory Instrument (MLLI) measures students’ expectations and experiences of the cognitive and affective domains of learning within the context of the “doing” of laboratory experiments. Each of the 30 MLLI items was coded a priori using the meaningful learning framework to be cognitive (e.g., I expect to focus on concepts, not procedures), affective (e.g., I worry about finishing on time), or a combination cognitive/affective (e.g., I felt unsure about the purpose of the procedures). MLLI was administered online via Qualtrics, and students were asked to indicate their agreement (from 0%, Completely Disagree to 100%, Completely Agree) with each statement. The MLLI was administered to students twice – 105 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

once at the beginning of the semester prior to completing any laboratory work to measure students’ expectations for learning, then again at the end of the semester to capture students’ learning experiences in the laboratory. The verbs in the items were changed to past tense for the end of semester administration. Data collection and analyses for multiple MLLI studies have been previously reported (12, 13, 19).

Results and Discussion

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

Inquiry Rubric Figure 3 depicts the level of inquiry across the 289 experiments evaluated in Study 1. Confirmation and Structured Inquiry were the two most common levels in both GC I and GC II courses, while Structured Inquiry and Guided Inquiry were most common in OC I. All experiments (N=10) in OC II were from one university and judged to be Guided Inquiry. None of the 289 experiments evaluated in Study 1 were at the level of Open or Authentic Inquiry, meaning that students were never asked to generate procedures or consider elements of experimental design, nor to pose their own question to investigate.

Figure 3. Levels of Inquiry across four types of courses in Study 1. Figure 4 shows how the level of inquiry varied across schools, ranging from 100% Confirmation (School 14, S14) to 80% Guided Inquiry (School 06, S06) in GC I. (Each school who volunteered to collect MLLI and/or FGS data was assigned a two-digit number. Not all schools are included in Figure 4 because some schools did not collect and/or return complete data sets.) For the five schools who provided 106 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

GC II experiments, the level of inquiry ranged from 58% Confirmation (School 24) to 40% Guided Inquiry (School 19). The five schools providing data for OC I were predominantly Guided Inquiry.

Figure 4. Levels of Inquiry across schools in Study 1.

Faculty Goals Survey GC and OC faculty responded similarly to each of the FGS items. Data for three items are included here as representative of their responses: “Laboratory activities and experiments selected for this course are designed to focus on skills that are transferable to research-oriented laboratories,” (Figure 5) “…have students present data in multiple formats,” (Figure 6) and “…teach students to build logical arguments based on their data” (Figure 7). 107 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

Figure 5. GC (left, N=22) and OC (right, N=12) faculty responses to FGS item that laboratories should be focused on skills that are transferable to research-oriented laboratories. (1 = strongly disagree, 6 = strongly agree).

Figure 6. GC (left, N=22) and OC (right, N=12) faculty responses to FGS item that the laboratory should be designed to have students present data in multiple formats, such as PowerPoint, posters, laboratory reports, etc.. (1 = strongly disagree, 6 = strongly agree).

108 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

Figure 7. GC (left, N=22) and OC (right, N=12) faculty responses to FGS item that the laboratories should teach students to build logical arguments based on their data. (1 = strongly disagree, 6 = strongly agree).

Meaningful Learning in the Laboratory Instrument (MLLI) In order to explore the data sets for both Study 1 and Study 2, one school was randomly selected (School 02) from amongst those that submitted FGS responses for two or more faculty, experiments for levels analysis using the rubric, and MLLI data for at least GC I and GC II. Plots of representative MLLI responses for GC I students (N=138) at School 02 can be found in Figures 8, 9, and 10. Pre-semester expectations are plotted on the x-axis and post-semester experiences on the y-axis. The diagonal line represents responses where experiences matched expectations. Points below the diagonal line represent responses where expectations exceeded experiences. In Figure 8, slightly more than half of the students (N=66) reported that their experiences with regard to making decisions about what data to collect failed to meet their expectations. In Figure 9, more than 75% of the students (N=108) reported that while they had expected to be excited about doing chemistry, their experiences failed to meet these expectations. In Figure 10, more than 73% of the students (N=101) reported that while they had expected to be required to interpret their data beyond only doing calculations, but their experiences failed to meet these expectations.

109 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

Figure 8. GC I students’ experiences vs. expectations for MLLI item 3 “to make decisions about what data to collect.”

Figure 9. GC I students’ experiences vs. expectations for MLLI item 8 “to be excited to do chemistry.”

110 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

Figure 10. GC I students’ experiences vs. expectations for MLLI item 22 “to interpret my data beyond doing calculations.”

Faculty Goals Survey and Inquiry Rubric In Study 1, the authors examined each of the FGS items to identify those that corresponded to the characterization of experiments as confirmation or inquiry experiences. Five items were identified and agreed upon by all the authors. Three FGS items corresponded to inquiry experiences in the laboratory: • • •

Teach students to build logical arguments based on data. The laboratory is designed to encourage the development of scientific reasoning skills. Laboratory is a place for students to learn to analyze data.

while two FGS items corresponded to confirmation experiences in the laboratory: • •

Explore concepts already discussed in lecture. The goal for laboratory instruction is to reinforce lecture content.

These five items were designated, respectively as Logic, Reasoning, Analyze Data, Concepts, and Lecture (Table 3).

111 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Table 3. Faculty Goals Survey responses for School 02 GC I Logic

Strongly Agree

Strongly Agree

Disagree

Strongly Agree

Agree

Strongly Agree

Concepts

Strongly Agree

Disagree

Lecture

Strongly Agree

Agree

Reasoning Analyze Data

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

GC II

To explore alignment between faculty goals and inquiry levels of experiences in laboratory experiments, data was analyzed for a randomly selected school from among those with FGS and Levels data for both GC I and GC II in the data set. The responses of the two faculty from School 02 on these five FGS items (Logic, Reasoning, Analyze Data, Concepts, Lecture) were summarized and compared to the levels of inquiry in the GC I and GC II experiments carried out at School 02. Table 3 and Figure 11 summarize the FGS and Levels data for School 02, respectively.

Figure 11. Levels of Inquiry in GC I and GC II for School 02.

Faculty Goals Survey and Meaningful Learning in the Laboratory Instrument In Study 2, because the FGS items and the MLLI items were all coded based on meaningful learning theory, we expected to identify cognitive or affective items that were identified as faculty goals and as elements of meaningful learning for students. Surprisingly, however, the items on the FGS could not be mapped reliably onto MLLI items, despite being coded as cognitive or affective by multiple researchers.

112 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Limitations

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

There are limitations to this study. We asked faculty to answer the FGS, provide their experiments, and their students to complete the MLLI twice in one semester – ideally for GC I, GC II, OC I, and OC II. Our data set consisted almost entirely of “partial” responses, e.g., faculty who did not answer the FGS, but sent experiments and whose students completed MLLI, or MLLI data for just one of the courses. This limited the data set of complete responses where FGS goals from a given instructor could be mapped to their selected experiments and their students’ MLLI responses.

Conclusions Figure 1 depicts a rational argument for how faculty goals for learning in the laboratory should drive experiment selection and ultimately students’ experiences. Two research studies were carried out to compare faculty goals with the degree of inquiry in their experiments (Study 1) and meaningful learning for their students (Study 2). In Study 1, the level of inquiry for the ‘same’ course varied across universities, with a general trend that as students moved from GC to OC, the degree of inquiry in their experiments increased. A comparison of Table 3 and Figure 11 reveals little consistency at School 02 between faculty goals and their selected experiments as some goals aligned with their selected experiments while others did not. In Study 2, while many of the students’ experiences failed to meet their expectations for cognitive and affective learning, we were ultimately unable to answer the original research question to examine how faculty goals for General Chemistry and Organic Chemistry laboratory are aligned with students’ expectations and experiences, despite using previously published data collection tools that generated reliable and valid data in previous studies. The FGS was developed from a voluminous corpus of interview data with faculty teaching General Chemistry, Organic Chemistry, and upper division (physical, analytical, biochemistry) laboratories at institutions whose laboratory program had remained unchanged for many years and at institutions that had successfully procured external funding to innovate their laboratory programs. The FGS items represent the consensus of chemistry faculty about what is important to learn in the undergraduate chemistry laboratory at a wide variety of institutions and courses. Meanwhile, the MLLI items represent cognitive and affective dimensions of the undergraduate chemistry laboratory consistent with theory about how human beings learn. The fact that these two methodological approaches resulted in non-overlapping data sets is not the result of poor research design, but rather an important piece of evidence that faculty approaches to choosing laboratory experiments for students are not aligned with opportunities for cognitive and affective learning . To further pursue the aims of Study 2, a new study was planned where faculty were asked to answer MLLI as they hoped their students would. Data analysis is underway and will be published in a future manuscript. 113 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Acknowledgments This work was supported by the Volwiler Family Endowment to the Miami University Department of Chemistry & Biochemistry and National Science Foundation grant number 0733642. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

References

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

1. 2. 3. 4.

5.

6.

7. 8.

9.

10.

11. 12.

13.

14.

Sacks, O. Uncle Tungsten: Memories of a Chemical Boyhood; Vintage Books: New York, 2002. Good, H. G. On the early history of Liebig’s laboratory. J. Chem. Educ. 1936, 13, 557–562, DOI: 10.1021/ed013p557. Hofstein, A.; Lunetta, V. N. The role of the laboratory in science teaching: Neglected aspects of research. Rev. Educ. Res. 1982, 52, 201–217. Hofstein, A.; Lunetta, V. N. The laboratory in science education: Foundations for the twenty-first century. Sci. Educ. 2004, 88, 28–54, DOI: 10.1002/ sce.10106. Hofstein, A.; Mamlok-Naaman, R. The laboratory in science education: The state of the art. Chem. Educ. Res. Pract. 2007, 8, 105–107, DOI: 10.1039/ B7RP90003A. Elliot, M. J.; Stewart, K. K.; Lagowski, J. J. The role of the laboratory in chemistry instruction. J. Chem. Educ. 2008, 85, 145–149, DOI: 10.1021/ ed085p145. Reid, N.; Shah, I. The role of laboratory work in university chemistry. Chem. Educ. Res. Pract. 2007, 8, 172–185, DOI: 10.1039/B5RP90026C. Rice, J. W.; Thomas, S. M.; O’Toole, P. In Tertiary Science Education in the 21st Century; Australian Council of Deans of Science: Melbourne, 2009; p 13. Bruck, A. D.; Towns, M. H. Development, implementation, and analysis of a national survey of faculty goals for undergraduate chemistry laboratory. J. Chem. Educ. 2013, 90, 685–693, DOI: 10.1021/ed300371n. Fay, M. E.; Grove, N. P.; Towns, M. H.; Bretz, S. L. A rubric to characterize inquiry in the undergraduate chemistry laboratory. Chem. Educ. Res. Pract. 2007, 8, 212–219, DOI: 10.1039/B6RP90031C. Bruck, L. B.; Bretz, S. L.; Towns, M. H. Characterizing the level of inquiry in the undergraduate laboratory. J. Coll. Sci. Teach. 2008, 37, 52–58. Galloway, K. R.; Bretz, S. L. Development of an assessment tool to measure students’ meaningful learning in the undergraduate chemistry laboratory. J. Chem. Educ. 2015, 92, 1149–1158, DOI: 10.1021/ed500881y. Galloway, K. R.; Bretz, S. L. Measuring meaningful learning in the undergraduate chemistry laboratory: A national, cross-sectional study. J. Chem. Educ. 2015, 92, 2006–2018, DOI: 10.1021/acs.jchemed.5b00538. Towns, M.; Bretz, S. L.; Bruck, L. B. Faculty perspectives of undergraduate chemistry laboratory: Goals and obstacles to success. J. Chem. Educ. 2010, 87, 1416–1424, DOI: 10.1021/ed900002d. 114

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch006

15. Bretz, S. L.; Fay, M. E.; Bruck, L.; Towns, M. H. What faculty interviews reveal about meaningful learning in the undergraduate laboratory. J. Chem. Educ. 2013, 90, 281–288, DOI: 10.1021/ed300384r. 16. Novak, J. D. Human constructivism: A unification of psychological and epistemological phenomena in meaning making. Inter. J. Pers. Const. Psych. 1993, 167–193, DOI: 10.1080/08936039308404338. 17. Bretz, S. L. Human constructivism and meaningful learning. J. Chem. Educ. 2001, 78, 1107, DOI: 10.1021/ed078p1107.6. 18. Cohen, J. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psych. Bull. 1968, 70, 213–220, DOI: 10.1037/h0026256. 19. Galloway, K. R.; Bretz, S. L. Using cluster analysis to characterize meaningful learning in a first-year university chemistry laboratory course. Chem. Educ. Res. Pract. 2015, 16, 879–892, DOI: 10.1039/C5RP00077G.

115 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Chapter 7

Exploring the Instructional Use of Contrasting Molecular Animations of a Redox Reaction Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

Resa M. Kelly* San José State University, San José, California 95136, United States *E-mail: [email protected]

Atomic level animations serve as a useful tool to help instructors communicate information about unseen atomic level processes to their students. They have typically been used as explanatory models to help students think about what the atoms, ions and molecules must be doing to account for macroscopic outcomes. The goal of this chapter is to provide an alternative teaching strategy for introducing animations that draws attention to key mechanistic features by showing animations that present contrasting depictions of a redox reaction event, in which the mechanisms are in variance to each other. For example, one of the animations offers a scientifically acceptable model of the reaction mechanism, while the contrasting animations offer scientifically unacceptable models. Students were asked to select the animation that they felt was the best representation and that also fit with macroscopic, experimental evidence. In this manner, the animation that the student selected served as a formative assessment check that revealed to the instructor what the student thought about the atomic level.

© 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Visualization Studies

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

“One cannot change a student’s mental model of chemistry concepts by simply showing them a different, albeit better, model in an animation (1).” In this work, animations refer to the dynamic computerized representation of the behaviors and interactions of submicroscopic entitites such as atoms and molecules during chemical reactions (1). They work by helping students to visualize chemical reactions at the submicroscopic level and to create imaginal representations of those reactions (1, 2). Many studies contend that students have a much better conceptual understanding of chemistry events after using molecular level animations (1–14). Learning is often reported to be improved, but inherent flawed conceptions persist as students adapt features observed in the visualization to fit with their previous concepts. Students attend to simplistic, general structural features and pay less attention to mechanistic details that they view as unnecessary details (14). One effective approach used in a number of studies has been to partner animations with observable macroscopic events such as demonstrations, simple laboratory activities or videos to provide students with an explanation of the observed phenomenon (1–14). In a study by Velázquez-Marcano, Williamson, Ashkenazi, Tasker and Williamson (2004) it was reported that animations by themselves were not as effective at promoting enhanced student performance as showing a molecular-level mechanism with the macroscopic process (7). Velázquez-Marcano et al. recognized that animations provide students with an opportunity to interpret a concrete phenomenon with the use of an abstract concept (7). However, in this study and the others previously mentioned, students were not asked to support or refute what they were seeing in the animation with experimental evidence. Instead, the students were asked to accept the explanation provided by the animation to make sense of the evidence. In essence, students were placed in the role of passive recipients of information, and they were expected to master the information and conform their understanding to fit with the features portrayed in the animations. This chapter proposes a teaching strategy to involve students in critiquing molecular level animations that are in variance to each other. The animations portray structural aspects of atomic level species or the particulate level involved in a redox reaction similarly, but they differ in how the reaction mechanism is portrayed, and they have different levels of scientific accuracy. By asking students to consider how the animations are supported or refuted by macroscopic experimental evidence, we gain better insight into how students understand the chemistry of the reaction event, and of course it is hoped that the students gain autonomy of their learning. Kelly and Jones (2007, 2008) provided deeper insight into how students’ understanding is affected by viewing animations partnered with a macroscopic event (9, 10). They studied how general chemistry students used two different animations to make sense of a simple laboratory activity in which solid sodium chloride was dissolved in pure water (9, 10). The students were asked to construct pictorial, oral and written explanations of the dissolution event at the macroscopic and submicroscopic levels before and after viewing two animations of varying 118 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

complexity of the same event. Their findings revealed that students incorporated some of the features viewed in the animations, but not all, and while students made obvious improvements, they retained misconceptions or sometimes developed new ones (9). More importantly, students had difficulty transferring what they learned into a new setting (10). Specifically, students who were asked to draw and explain what the atomic level of an aqueous solution of sodium chloride looked like, before it was mixed with aqueous silver nitrate to form a precipitate, regressed to previous conceptions unless they were prompted to recall what they had learned from the animations. This was an early indication that using the animations as an explanation was less effectual than hoped; however, it was reassuring to note that the students could recall what they saw and only needed a reminder to assist in their recollection. Tasker set the stage for studies on a simple redox reaction when he and his VisChem cadre designed an animation that conveyed the dynamic and multi-particulate nature of the laboratory redox event in which silver crystals were grown on the surface of copper metal to show the reduction of many silver ions on the copper surface, with the simultaneous release of half as many copper(II) ions from the metal lattice (1). When students draw their particulate level understanding and view animations they recognize a greater number of key features and develop vivid mental imagery of the phenomena (1). Rosenthal and Sanger (2012) contributed to the research field on redox chemistry and learning from animations when they examined how students took in information after first viewing a chemical demonstration of the redox reaction event between silver ions and copper atoms (11, 12). The students in their study were shown a demonstration in which “solid silver nitrate was dissolved in water; after the solid dissolved, a piece of solid copper metal was added to the solution and allowed to react.” The participants were then asked to observe one of two animations and orally explain how their perceptions of the reaction changed based on viewing the animation. The students were shown two different animations, both considered accurate representations, of the same redox reaction that emphasized different aspects of the reaction and also portrayed different levels of complexity. They noted that students misinterpreted what they saw and sometimes it was revealed that students had misconceptions that may have led to misinterpretation of events observed in the animation. Their findings indicated that students had more difficulty interpreting the more complex animation (VisChem) than their own redox animation (12). They concluded that students had difficulty interpreting two different animations of differing complexity. However, having difficulty interpreting animations may be advenantageous to the learning process. It creates an opportunity to challenge students to reflect on their beliefs and understanding. A pilot study was conducted to learn how first semester general chemistry students responded to two animations in conflict to each other that served as possible atomic level models for describing a video-recorded redox reaction between solid copper metal and aqueous silver nitrate. One of the animations was accurate in its depiction and also a popularly used animation from VisChem (1), while the other animation was one that an artist from San José State University constructed in collaboration with the author to be an inaccurate representation. Fom here on it will be referred to as the SJSU animation in reference to its origin. 119 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

The findings of the study indicated that students improved their understanding of the atomic level arrangement of copper atoms and that silver atoms deposited on its surface. However, students continued to struggle with how to represent aqueous salt solutions before and after the reaction and with the reaction mechanism in which electrons were transferred from copper to silver resulting in neutral silver atoms coating the surface of the copper and the copper ions going into solution. As a result of this study, three new animations of the redox reaction between aqueous silver nitrate and solid copper were designed. The two animations from the pilot study were redesigned to be stylistically similar. In the initial study, as mentioned, efforts were taken to create an animation that was a “wrong” version of the VisChem animation, but unfortunatley the animations were quite different from each other stylistically and in length of time. Some students thought that the SJSU “wrong” animation was merely a simplified version of the VisChem animation. By making the animations similar in structural details apart from the reaction mechanism, the animation designers felt they would be better positioned to study students’ reaction mechanism preference without confounding their selection for stylistic reasons. Another finding from the pilot study, was that nearly all of the students’ initial drawings, made prior to viewing animations, did not fit with the depiction in the animations. Most of the students initially thought that all molecules, atoms or ions of the aqueous solution plated onto the solid wire surface. Since the other two animations did not depict this, a third animation was created to represent this mechanism and to see if students might be more inclined to choose an animation that fit best with this common misconception. While the animation research studies described are in active stages of completion, the goal of this chapter is to present an example for how these animations could be used in actual class instruction. Specifically, the purpose of this chapter is to share how students responded when presented with the challenge to select the animation that best fit with the experimental evidence. The learning goal presented to students was: “How do you use physical evidence to predict what a reaction looks like at the atomic level?”

The Lesson Challenge: Choose an Animation That Best Fits with Experimental Evidence Participants The diversity of the population of students that were enrolled in this General Chemistry course was likely consistent with the mid-sized western university from which they attended, which has a wide distribution of ethnicities: 32% Asian, 23% Hispanic, 22% White, 3% African American, 11% Foreign Nationals and 9% Others, although this was not examined. Not all students enrolled in the class attended on the days the exercise was introduced. On the first day, 178 out of 218 students enrolled attended based on question-and-answer polling, also known as REEF EducationTM polling responses and 163 students attended the second day of the exercise. The gender make-up was not determined. The author of this chapter designed the video and animations and instructed the class. 120 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

Prior to the video/animation exercise, students were instructed about the nature of aqueous salt solutions with handmade drawings of the particulate nature of matter, as well as how to represent the solutions symbolically. They were also taught that they could determine if ions were present in solution by testing aqueous solutions with a hand-held conductivity tester. If the conductivity tester lit up, then the students were told that they could be confident that free, mobile ions were present in solution and if it lit up weakly, assuming concentration was held constant, then they could conclude that the substance did not dissociate well into ions. If the solution produced no light, then they could assume it was composed of nonelectrolytes. The students completed a lab experiment on conductivity, in which a variety of substances were tested using the same handheld conductivity tester depicted in the video, and they wrote net ionic equations to describe solutions that contained strong, weak and non-electrolytes. They had practice writing three kinds of equations: molecular equations, total ionic or complete ionic equations, and net ionic equations, as mentioned previously. They also completed a lab on precipitation reactions. They were shown animations on precipitation reactions, also designed by the instructor, and they were given access to a website where they could further review the animations. The students were tasked with drawing representations of solid and aqueous sodium chloride in class and the instructor frequently drew atomic level representations on the marker board to discuss reactions. Leading into this video/animation exercise, the students were shown illustrations of macroscopic evidence of several reactions between metals and aqueous salt solutions. They were instructed on how to predict whether a reaction would occur by using an activity series and they practiced writing the three kinds of equations. The instructor also modeled total ionic equations as particulate level representations on the marker board and discussed the process of reduction and oxidation and how to distinguish a redox reaction from a precipitation reaction by use of oxidation numbers. The Video/Animation Exercise Upon completing the discussion on redox reactions between metals and aqueous salt solutions, the instructor presented students with a challenge. She showed them a video of an experiment demonstrating the reaction between solid copper metal and aqueous silver nitrate (Figure 1). In the video, pure water was mixed with a small scoop of silver nitrate to make an aqueous silver nitrate solution that tested positive for electrical conductivity, and the same procedures were followed for making a copper (II) nitrate solution that also tested positive for electrical conduction. Pure water was added to one test tube, the aqueous silver nitrate solution was added to the next test tube and the aqueous copper (II) nitrate was added to the third test tube. A coiled copper wire was added to each test tube. Time elapsed for eight minutes before each wire was removed and the resulting solutions were once again tested for conductivity. A grey substance formed on only the wire placed in the aqueous silver nitrate solution. The grey substance was scraped off the wire and revealed copper wire underneath the grey solid. The resulting product solution changed to a pale blue color that matched the color of the copper(II) nitrate solution that served as a control, and also tested positive for 121 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

electrical conduction. The copper (II) nitrate solution did not react with the wire and tested positive for electrical conduction both before and after the wire was introduced with no apparent change. The pure water did not react with the wire, and it did not conduct when tested before or after the wire was introduced.

Figure 1. A still image from the video showing an aqueous silver nitrate solution being made.

After the video played only once, the students were instructed that they would next view three different animations that could serve as possible atomic level explanations for the reaction that took place between the solid copper metal and aqueous silver nitrate. They were cautioned that not all of the animations were correct and it was their job to figure out which one best fit with the evidence that was presented in the video of experimental evidence. Before an animation played, the instructor identified each species that appeared on the opening picture of the animation.The first animation, unbeknownst to the students, was a wrong model in which aqueous silver nitrate molecules floated about in solution, and then began to travel toward the copper surface, separated and then came to rest on the copper surface (Figure 2). The animation was made using Maya Computer Animation & Modeling Software, and it was seven seconds long. The students were informed that water molecules were removed to simplify the view, but they should trust that water molecules were present. The second animation, unbeknownst to the students, was the most accurate model portraying the electron transfer mechanism (Figure 3). This animation was also made using Maya software, and it was ten seconds long. Once again students were instructed that most of the solvent water molecules were removed except for the ones hydrating the ions to ease the viewing experience. The animation began with a purple silver ion moving toward the copper surface. It then gained an electron cloud from the electron transfer. Next, a second silver ion was drawn to the copper surface. It gained an electron cloud as it was reduced. Nearby a copper atom was oxidized and a copper ion formed attracting water molecules, which drew the copper ion away from the lattice. The nitrate ions remained inactive and floated aimlessly in solution. 122 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

Figure 2. a) Animation 1 portrayed silver nitrate molecules in solution next to a lattice of copper atoms. b) The silver nitrate molecules separate and attract to the copper surface. c) All solute species end up on the surface of the copper metal. The water molecules were removed for simplification.

Figure 3. a) Animation 2 shows an aqueous solution of silver and nitrate ions with a copper lattice and an electron cloud overlay. b) The silver ion migrates toward the copper lattice and a cloud forms around it. c) Another silver ion is converted to a neutral atom at the surface, while nearby a copper ion is revealed and water molecules attract to it. d) The water molecules pull the copper ion into solution. 123 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

The third animation, unbeknownst to the students, was a wrong model designed to fit with the look of a single replacement equation (Figure 4). This animation was also made using Maya software, and it was eight seconds long. The students were informed that once again, solvent water molecules were removed to ease the viewing experience. The animation began with silver nitrate molecules floating in solution. One molecule collided with the copper surface and broke apart, followed by another molecule, which did the same. The detached nitrates then attracted to a copper atom and pulled it into the solution space.

Figure 4. a) Animation 3 shows silver nitrate molecules in solution with a copper lattice. b) The silver nitrate molecule collides with the copper surface and breaks apart. c) Another silver nitrate molecule collides with the copper surface, and the two nitrates attract to a copper atom in the lattice. d) The nitrates pull the copper atom into solution. The students worked by themselves, in pairs or in groups of 3 and 4 students by their choice as this was how the instructor conducted her classroom practice. While students discussed their options they sometimes asked the instructor to replay an animation. Thus, each animation was played one additional time. Next, each student voted with the mobile classroom response system, REEF EducationTM polling, for the animation that they felt best fit as an explanation of the video of experimental evidence. The outcome of the REEF EducationTM polling, that was shown to the students, indicated that 79% (141 of 178) of the students who voted, chose animation 3 as the best fit, while 20% (36 of 178) voted for animation 2 and only 1% (1 of 178) voted for animation 1. As mentioned, the students were shown the vote outcome, but they were not informed of the correct answer, which was animation 2. Since the outcome of the students’ poll was unexpected and not at all desired, the instructor wanted to provide an additional opportunity for the students to revisit the animations and give 124 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

an explanation for why they selected the animation that they did. The next time they met for class, two days after the initial presentation of the animations, students were reminded of the animation exercise, and they were instructed that they were going to practice making explicit connections between the experimental evidence and their animation choice. The instructor did not reveal which animation was best. Specifically, they were asked to focus on three key pieces of evidence (Figure 5). First, they were orally asked: How did the animation they chose account for the conductivity result? The questions on the slide were posted to serve as a guide while students worked out their explanation, however, the instructor practiced a conversational style with her students and would orally embellish discussion with more than what appeared on the slide. The instructor pointed to the two photos on the slide showing solutions being tested for conductivity with a hand-held conductivity tester device as a visual reminder (Figure 5). Students were familiar with these types of testers as they were modeled for them in their second laboratory session. It was pointed out that the aqueous silver nitrate solution conducted before the reaction and the resulting product solution conducted after the reaction. Second, students were asked: Did your animation choice account for the change that appeared at the surface of the wire? The instructor pointed to the picture on the slide showing the coated copper wire (Figure 5). Third, how does your animation choice account for the blue color of the final solution? Once again, the instructor referred to the still image picture showing the blue solution (Figure 5). Students were reminded that in their written justification, they should explain their answers, not just respond yes or no, and they were once again allowed to work in groups. During the discussion time of approximately 15 minutes, students asked the instructor to replay each animation, thus each animation was played two to three times during the discussion time. In total there were 75 groups with the label group referring broadly to any individual or pairing of 2 or more students. Students were asked to have one person write the answer to the three questions, and they were informed to reach consensus on their answers and to indicate which animation they were defending. If they worked alone, which was rare, they simply reported based on their solitary understanding. After students submitted their justifications, they were given the opportunity to individually vote for the animation that they felt was best in case their justification caused them to change their animation choice. After the revote, the outcome indicated that 1% (2 of 163) voted for animation 1, 40% (66 of 163) voted for animation 2, and 58% (95 of 163) voted for animation 3. After sharing the voting bar graph with the class the instructor revealed that the most accurate animation was animation 2, and she informed students that she was interested to learn why they made the selection that they did. One student responded that animation 3 best fit with the formulas in the balanced equation and this persuaded his group to pick this animation. Another student shared that in animation 2, the copper atom changed to become blue in color and his group assumed that this meant a new element formed, not an ion. He remembered that when learning to balance equations they were taught the law of conservation of matter and matter would not be created. For this reason, he felt animation 2 must be incorrect and chose animation 3. At this point, the class had reached the end and students were dismissed. 125 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

Figure 5. Power point slide shown in class instructing students to connect the animations to the experimental evidence.

Justifying the Connection between Three Key Experimental Features and the Animations As mentioned previously, the students were tasked with providing a written justification for the animation they selected in terms of its fit with the three key features: conductivity evidence, the change to the copper surface and the product solution’s blue color. This next section examines how students responded to this exercise and how they decided that the animation they chose as a best fit with the experimental evidence was justified based on the three key pieces of evidence. Since only two students preferred animation 1, and neither student wrote a justification for this choice, this animation was not reviewed for its justification.

Animation 2 – The Animation That Represents Redox Most Accurately Sixty-six students (out of 163) chose animation 2, the most accurate animation, as having the best fit with experimental evidence. This was an improvement over the first time they made their animation selection when only 36 of 178 chose it. Thirty-two groups provided written justifications for animation 2’s fit with the three key experimental features. The level of sophistication of the justification varied greatly (Tables 1-3). Students develop fragmented and incomplete understanding and drawing wrong conclusions may be a necessary step in the learning process (15). In this case, the unsophisticated justification statements reflected students’ fragmented and incomplete understanding. In addition, this was the first time the students had been asked to complete such an exercise in which they had to justify an animation choice which may have affected the quality of their statements too.

126 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

Table 1. Analysis of Students’ Justification for Choosing Animation 2 in Connection to the Conductivity Evidence Examples that Capture Students’ Justification

Does the animation account for the conductivity evidence? Explanation

Student Group Responses 32 total groups, n= 66

None

5

Answers yes, but reasoning lacks connection between macroscopic and submicroscopic levels.

3

- Cu(NO3)2 was conductive. - AgNO3 is conductive and the copper residue in the silver nitrate still allowed conductivity

Answers yes, but reasoning indicates inappropriate or weak connection between the macroscopic and submicroscopic levels.

14

- The 3rd animation had-free floating ions that would be conductive. The copper and silver swapped places in the third animation forming Cu(NO3)2 -The Cu molecules pulled by the water were allowing conductivity

Answers yes, reasoning suggests an appropriate connection between the macroscopic and submicroscopic levels.

10

- To be able to conduct the AgNO3(aq) needs to break into ions and only animation 2 was showing there are ions and after the reaction, ions of Cu also stay apart from the NO3

Conductivity Evidence Students were asked: “Does the animation account for the conductivity evidence taken before and after the reaction?” Several (5 groups) selected animation 2, but gave no justification of whether the animation fit with the conductivity evidence. Only three groups responded that yes, this animation justified the experimental evidence, but their explanations were not very compelling as they lacked connection between the macroscopic and submicroscopic levels. Fourteen groups provided a justification, but they tended to be vague or their ability to connect the macroscopic level to the submicroscopic levels was weak and sometimes they misused terms like atom, ion and molecule. Ten groups gave answers that described a reasonable connection between the atomic level and macroscopic levels, largely recognizing that conductivity required free ions and animation 2 portrayed separated ions in solution before and after the reaction, thus it fit.

127 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

Table 2. Analysis of Students’ Justification for Choosing Animation 2 in Connection to the Copper Surface Change Examples that Capture Students’ Justification

Does it account for the change at the surface? Explanation

Student Group Responses 32 total groups, n=66

None

6

Answers yes, but reasoning lacks connection between macroscopic and submicroscopic levels.

10

- The color of the wire is a bit darker - The color of the copper is different, it seems more corroded.

Answers yes, but reasoning indicates inappropriate or weak connection between the macroscopic and submicroscopic levels.

9

- Ag(aq) moves to react with Cu(s) and creates a compound of Cu(NO3)2. - The surface will become darker since copper is taken out. - Yes because the two elements combine and form a strong bond and since both connect there is no loss of electrons.

Answers yes, reasoning suggests appropriate connection between the macroscopic and submicroscopic levels.

7

- The silver ion sticks to the surface of the metal. That accounts for the color change at the surface of the wire. - Yes, the silver ions are bonding to the outside of the copper.

Copper Surface Change The second feature students were asked to justify was: Does the animation account for the change at the surface of the copper metal? In animation 2, for every two silver ions that were reduced, one copper ion was oxidized and was drawn into solution. Classification of students’ justification for selecting animation 2 is presented (Table 2). Students who chose this animation typically responded that yes, the surface change was depicted, but they struggled to explain how (10 groups). In several cases, the explanation was incorrect and it did not seem as though the students understood why the surface change occurred. Fortunately, there were seven groups that made a connection between the macroscopic and submicroscopic levels; however, most of these did not address that the silver was reduced. Instead, they focused on the movement of the silver ion toward the copper lattice and described the bonding of the silver. This evidence may indicate that the animation may be too subtle in its portrayal of electron transfer.

128 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Appearance of Blue Product Solution

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

The last feature that students were asked to justify in connection to the animations was how the blue solution formed. In animation 2, the copper atom was oxidized and the water molecules carried the ion into solution. It is the hydrated copper ion that is responsible for the blue solution color. The students’ responses revealed a range of sophistication (Table 3). Several groups (4) did not provide justification, and many groups (6) gave weak or faulty reasoning for how the animation accounted for the blue solution, suggesting nitrates were involved or that it was the result of a compound. Eleven groups made the appropriate connection that the copper ion was responsible for the blue colored solution.

Table 3. Analysis of Students’ Justification for Choosing Animation 2 in Connection to the Blue Color of the Product Solution Examples that Capture Students’ Justification

Does it account for the blue color of the product solution? Explanation

Student Group Responses 32 total groups, n=66

None

4

Answers yes, but reasoning lacks connection between macroscopic and submicroscopic levels.

10

- Cu is higher than silver on activity series and easily oxidized. - The blue color of the final solution changed.

Answers yes, but reasoning indicates inappropriate or weak connection between the macroscopic and submicroscopic levels.

6

- Once the nitrate combined with the copper the copper atom appeared blue, which would explain why the solution turns blue in the end. - It also created a new compound, which can account for the blue color.

Answers yes, reasoning suggests appropriate connection between the macroscopic and submicroscopic levels.

11

- It accounts for the blue coloring because once Ag+ connects some water is seen pulling a blue copper away. - The water is blue because copper ions separate into the water. Cu(NO3)2(aq) is blue in color.

129 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Animation 3 –The Animation Resembling the Reaction Equation

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

Animation 3 (Figure 4), one of the wrong models, was the most popular choice by students with 141 of 178 students initially choosing this animation as the best representation. However, after students were asked to justify the animation’s connection to the evidence, the number of students believing this animation to be the best fit dropped to 95 of 163 students. Forty-three groups submitted justification for choosing animation 3. There was also considerable variation in the level of sophistication of their written justifications (Table 4-6).

Table 4. Analysis of Students’ Justification for Choosing Animation 3 in Connection to the Conductivity Evidence Examples that Capture Students’ Justification

Does the animation account for the conductivity evidence? Explanation

Student Group Responses 43 total groups, n=95

None

11

Answers yes, but reasoning lacks connection between macroscopic and submicroscopic levels.

6

- Cu(NO3)2 was conductive. - AgNO3 is conductive and the copper residue in the silver nitrate still allowed conductivity

Exchange or switching lead to conduction

13

-The 3rd animation had free floating ions that would be conductive. The copper and silver swapped places in the third animation forming Cu(NO3)2

Conduction is due to the copper.

2

- Cu in aqueous form and Cu is a metal so it should be conductive.

Presence of free ions

4

- There are still free ions conducting electricity.

Justification does not match with the animation features

6

- When a molecule comes in contact with water if it breaks apart then that aqueous solution is conductive.

Justification does not match with experimental evidence

1

- It was not conductive in the beginning because there were molecules but few ions, after the reaction there were now ions, and conductivity increased.

130 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

Conductivity Evidence In connection to the conductivity evidence, eleven groups who selected animation 3 gave no response to account for how the animation was justified based on the experimental video evidence and six groups responded yes, but did not elaborate or were unable to make a connection between the macroscopic and submicroscopic levels. To summarize, 17 of the 43 groups were unable to satisfactorily justify why they chose the animation in connection to the conductivity evidence. Interestingly, thirteen groups described the process whereby the copper and silver swapped places or exchanged ions as the connection to conductivity. Four groups stated that free-floating ions would be conductive and the free ions were present when the ions were exchanged. These groups failed to recognize that the conductivity was tested before and after the reaction, not during the reaction. It is important to acknowledge that the students understood that conductivity involved free ions, and they found a way to fit the animation model’s features to make a reasonable guess as to how this model might work. Although wrong in their choice, they have modeled that they have learned how to connect evidence to models to justify their scientific reasoning, a positive attribute. Only 2 groups thought that copper was responsible for the conductivity as it was a metal and metals conduct. A small number of groups (3) wrote justifications that either did not seem to fit with what the animation depicted or did not fit with the conductivity evidence.

Copper Surface Change In animation 3, when two silver nitrates collided with the copper surface, the nitrate ions detached and bonded to a copper atom on the surface, leaving behind two silver atoms. Students were asked: Does the animation account for the change at the surface of the copper metal? Classification of students’ justification for how animation 3 accounted for the change at the copper surface is presented in Table 5. Seven groups did not provide a justification and two groups replied that they did not feel that the animation provided a justification consistent with the evidence. Ten groups reported that there was a change to the surface, but they provided little explanation, typically only confirming that the surface changed. Several groups (6) made inappropriate connections or constructed a justification that did not fit with the information portrayed in the animation. Many students (15 groups) made a logical connection noticing that the silver could have been deposited during the exchange, which would be consistent with the animation and a good fit with the evidence.

131 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

Table 5. Analysis of Students’ Justification for Choosing Animation 3 in Connection to the Copper Surface Change Examples that Capture Students’ Justification

Does it account for the change at the surface? Explanation

Student Group Responses 43 total groups, n=95

None

7

Answers no, no justification

2

Answers yes, but reasoning lacks connection between macroscopic and submicroscopic levels

10

- The surface of the wire does change - Ag(s) must have formed on the surface.

Answers yes, but reasoning indicates inappropriate or weak connection between the macroscopic and submicroscopic levels.

6

- Wire had a substance caked on, which was shown with the nitrate ions sticking to the copper. - Yes it does the change at the surface of the wire happens because of the reaction.

Answers yes, and describes silver or a substance being deposited due to the reaction/ exchange.

15

- Ag+ ion attached onto copper wire for every copper molecule being taken away, so it does change on surface. - Because the AgNO3 and the copper reacted causing the silver to be deposited on the spring

Justification does not match with the animation features

3

- Yes copper loses electron, causing wire to rust.

Appearance of Blue Product Solution The last feature that students were asked to justify in connection to the animations was how the blue solution formed. In animation 3, the copper atom/ion was drawn away from the lattice by two nitrate ions that originally carried the silver toward the copper lattice before releasing the silver at the surface (Figure 4d). The logical explanation, adopted by eight groups, was that copper (II) nitrate was responsible for the blue color. Most groups (13 groups) attributed the blue color to reaction dynamics focusing on the copper being extracted (5 groups), the silver being removed from solution (2 groups), or the general exchange of ions (6 groups) as responsible for the blue color. Seven groups gave weak reasons that were often vague, likely because the students were uncertain of the cause of the blue color. Six groups gave no explanation and three groups responded that the animation did not provide a justification for the blue solution color.

132 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

Table 6. Analysis of Students’ Justification for Choosing Animation 3 in Connection to the Blue Color of the Product Solution Examples that Capture Students’ Justification

Does it account for the blue color of the product solution? Explanation

Student Group Responses 43 total groups, n=95

None

6

Answers no, no justification

3

Answers yes, but reasoning lacks connection between macroscopic and submicroscopic levels

7

- The water was left blue so something had to be left over.

Answers yes, but reasoning indicates inappropriate or weak connection between the macroscopic and submicroscopic levels.

5

- Blue water represents Cu in aqueous form. After reaction we’re left with aqueous solution containing Cu Ag and NO3 - Yes because Cu + AgNO3 turn the white liquid into a blue liquid.

Answers yes, justification focuses on movement of ions, extraction, or transfer

13

- Yes because the nitrates show it causes the copper to leave its solid state to become part of the aqueous solution and be a blue solution. - It is blue because of the change of ions after reaction.

Answers yes, copper containing molecules cause the blue color.

8

- Yes, because copper is blue. The blue color solution means that the reaction occurred and copper nitrate formed.

Limitations This chapter presents an authentic teaching practice. As a result there are a few limitations that should be considered as you review the outcome of this teaching practice. First, the students were not placed into structured groups and the influence of group dynamics was not investigated. Second, the students may have discussed the animations outside of class or they may have worked with different partners for the second session. Thus, one cannot conclusively state that students revised their animation selection based only on being tasked to defend their animation choice with its fit to the experimental evidence. As the instructor did not disclose that she would revisit the animations in the next class, students were likely not motiviated to reconsider their selection in preparation for class.The purpose of revisiting the animations was to find out how the students connected to the animations and how they justified their choice. Third, the instructor, who is the author of this paper, also designed the animations. She was aware of her bias toward knowing that animation 2 was the best choice, but she was careful to not reveal this. Based on the outcome, it seems that her knowledge of the 133 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

best animation did not affect the outcome in the first phase, but perhaps students detected that something was awry by being asked to revisit the animations a second time affecting their selection in phase 2.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

Conclusions In this study, animations that were in variance in their mechanistic portrayal of a redox reaction were used to examine how students determined their scientific legitmacy in connection to experimental evidence. This instructional strategy holds the potential to develop skills necessary for critiquing models, and it gives students the freedom to draw the wrong conclusion which may be a necessary step in the learning process (15). Student knowledge reorganization may playout in different ways when they take on conceptually challenging tasks. This exercise provides students the opportunity to practice critiquing the plausibility of animations as they fit with experimental evidence. The video/animation exercise in this study served as a formative assessment tool, in that it helped the instructor quickly evaluate students’ atomic level preference as deduced from the students’ REEF EducationTM polling response. From this outcome, the instructor made a decision on how to modify her instruction and came up with an additional follow-up exercise to have students articulate why the animation features fit with the experimental evidence. The justification exercise revealed that students needed more practice learning to critique models in connection to experimental evidence. As a result of this exercise, it was noticed, albeit anecdotally, that students paid closer attention to visual representations in the questions that they asked during office hours, as they were worried that they would be tested over pictorial representations. In fact, being wrong in their animation selection was rather shocking to the students and this seemed to entice some students to reevaluate how they thought about the redox reaction. Of course it must be acknowledged that the exercise was a novel task. Even though students regularly used REEF EducationTM polling and were asked to share their thoughts with a nearby partner, the practice of critiquing animations in comparison to evidence was novel and that alone may have resulted in deeper more thoughtful reflection. In this study, the students were informed of the correct answer, but in scientific research, we must rely on peer review as the litmus test for whether our findings hold merit. Perhaps as more instructors adopt this strategy for introducting animations, it will be less critical to inform students of whether they are right or wrong and instead students can be engaged in debate to convince each other of the accuracy of their selection. Tools such as this video/animation exercise empower students to critique models of chemistry events, which should help them to critique other models rather than simply accept them as facts. This exercise alone may be the greatest contribution that we can make toward advancing students to think like scientists and it may also help them to think more critically about scientific information communicated to them in their daily lives.

134 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Acknowledgments

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

The author wishes to acknowledge that the National Science Foundation under Grant No. 1525557 supported this work. Any opinions, findings, and conclusions or recommendations expressed in this chapter are those of the author and do not necessarily reflect the views of the National Science Foundation. The author also wishes to thank animation artist – Mina Evans who designed all of the animations shared in this chapter. Finally, special thanks to the team of Sevil Akaygün, Sarah Hansen, and Adrian Villalta-Cerdas for their assistance with the design of the video and animations.

References 1.

Tasker, R; Dalton, R. Research into practice: visualisation of the molecular world using animations. Chem. Ed. Res. Pract. 2006, 7, 141–159, DOI: 10.1039/B5RP90020D. 2. Yang, E.; Andre, T.; Greenbowe, T. J.; Tibell, L. Spatial ability and the imact of visualization/animation on learning electrochemistry. Int. J. Sci. Educ. 2003, 25, 329–349, DOI: 10.1080/09500690210126784. 3. Williamson, V. M.; Abraham, M. R. The effects of computer visualization on the particulate mental models of college chemistry students. J. Res. Sci. Teach. 1995, 32, 521–534, DOI: 10.1002/tea.3660320508. 4. Sanger, M. J.; Greenbowe, T. J. Common student misconceptions in electrochemistry: galvanic, electrolytic, and concentration cells. J. Res. Sci. Teach. 1997, 34, 377–398, DOI: 10.1002/(SICI)10982736(199704)34:43.3.CO;2-E. 5. Sanger, M.; Phelps, A.; Fienhold, J. Using a computer visualization to improve students’ conceptual understanding of a can-crushing demonstration. J. Chem. Educ. 2000, 77, 1517–1520, DOI: 10.1021/ ed077p1517. 6. Wu, H.; Krajcik, J.; Soloway, E. Promoting understanding of chemical representations: students’ use of a visualization tool in the classroom. J. Res. Sci. Teach. 2001, 38, 821–842, DOI: 10.1002/tea.1033. 7. Velázquez-Marcano, A.; Williamson, V. M.; Ashkenazi, G.; Tasker, R.; Williamson, K. C. The use of video demonstrations and particulate visualization in general chemistry. J. Sci. Ed. Technol. 2004, 13, 315–323, DOI: 10.1023/B:JOST.0000045458.76285.fe. 8. Ardac, D.; Akaygun, S. Effectiveness of multimedia-based instruction that emphasizes molecular representations on students’ understanding of chemical change. J. Res. Sci. Teach. 2004, 41, 317–337, DOI: 10.1002/tea.20005. 9. Kelly, R. M.; Jones, L. L. Exploring how different features of animations of sodium chloride dissolution affect students’ explanations. J. Sci. Educ. Technol. 2007, 16, 413–429, DOI: 10.1007/s10956-007-9065-3. 10. Kelly, R. M.; Jones, L. L. Investigating students’ ability to transfer ideas learned from molecular visualizations of the dissolution process. J. Chem. Educ. 2008, 85, 303–309, DOI: 10.1021/ed085p303. 135 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch007

11. Rosenthal, D. P.; Sanger, M. J. Student misinterpretations and misconceptions based on their explanations of two computer animations of varying complexity depicting the same oxidation-reduction reaction. Chem. Ed. Res. Pract. 2012, 13, 471–483, DOI: 10.1039/C2RP20048A. 12. Rosenthal, D. P.; Sanger, M. J. How does viewing one computer visualization affect students’ interpretations of another visualization depicting the same oxidation-reduction reaction? Chem. Ed. Res. Pract. 2013, 14, 286–296, DOI: 10.1039/C3RP00006K. 13. Akaygun, S.; Jones, L. Research-based design and development of a simulation of liquid-vapor equilibrium. Chem. Educ. Res. Pract. 2013, 14, 324–344, DOI: 10.1039/C3RP00002H. 14. Kelly, R. M. Using variation theory with metacognitive monitoring to develop insights into how students learn from molecular visualizations. J. Chem. Educ. 2014, 91, 1152–1161, DOI: 10.1021/ed500182g. 15. Øyehaug, A. B.; Holt, A. Students’ understanding of the nature of matter and chemical reactions – a longitudinal study of conceptual restructuring. Chem. Ed. Res. Pract. 2013, 14, 450–467, DOI: 10.1039/C3RP00027C.

136 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Chapter 8

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

Improving Students’ Practical Laboratory Techniques through Focused Instruction and Assessment John P. Canal,*,1 Jimmy Lowe,*,2 and Rosamaria Fong2 1Department

of Chemistry, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada V5A 1S6 2Department of Chemistry, British Columbia Institute of Technology, 3700 Willingdon Avenue, Burnaby, BC, Canada V5G 3H2 *E-mails: [email protected] (J.P.C.); [email protected] (J.L.)

Undergraduate chemistry laboratory courses are delivered through hands-on exploration of the lecture material, however, students face challenges in performing the physical actions of the experiments and these are dependent on the method the instructions are delivered. A student who knows and understands proper laboratory techniques will have an easier time with the experiment, be less stressed, work more safely, obtain results that match expectations and re-enforce the lessons learned in lecture. Recently, we have re-evaluated and modified the way the laboratory technique instruction is delivered to students in our laboratory courses. In order to focus the attention of students on the proper way to use the glassware and common apparatus used in most undergraduate laboratories, a Laboratory Techniques experiment was developed as well as laboratory technique centered exercises. We will present our efforts to improve student learning, instructor observations, data and graphing skills to support the effectiveness of these initiatives.

© 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

Introduction The role of the undergraduate chemistry laboratory is to provide students with an opportunity to explore theory through physical manipulations, while developing proper laboratory technique skills. To maximize learning, students need to develop proper laboratory techniques, as poor techniques will result in poor data, which will cause confusion when students try to link theory to practice. One of the first major works that highlighted the pedagogical importance of learning proper laboratory techniques was Michael Faraday’s Chemical Manipulation, written in 1827 (1). In this work Faraday wrote that “The importance of instruction in manipulations has long been felt by the author and the deficiency existing in the means of teaching it (1).” The result of Faraday’s work was an acceptance of laboratory technique instruction as an important component of the laboratory experience (2, 3). The methods employed by instructors to impart to students laboratory skills have varied greatly since the initial works by Faraday and are the subject of numerous publications (2–7). One of the more common methods employed is the “elbow instruction” or more commonly known as “feedback”, where an instructor observes a student performing a task in the laboratory and provides instructions on how to improve his/her technique (2). Our study is based on the work undertaken at the Department of Chemistry at Simon Fraser University (SFU) and the British Columbia Institute of Technology (BCIT). The entire BCIT Chemical and Environmental Technology diploma program was recently revised. This was an opportunity to change how first year laboratories could be taught more effectively to students. The updated BCIT CHEM 2204 laboratory course lead to a collaboration with SFU who were carrying out a course review of how laboratory techniques were taught in CHEM 126. Both courses are the second laboratory course in the students’ programs. In our attempt to improve our students’ learning experience in the laboratory we expanded our laboratory technique instructions based on the “feedback” method. Through students’ improved results, we will provide evidence of the effectiveness of our approach. The emphasis on the practical and transferable skills aids the students to prepare for future coursework and employment. We discuss some of the differences at both institutes in dealing with class size, resources and the use of teaching assistants. For example, the longer second-term at BCIT, provides an opportunity to evaluate students using Microsoft Excel for data manipulation and graphing.

Approaches to Laboratory Techniques Development Simon Fraser University (SFU) The first year chemistry laboratory courses at SFU follow the standard expository (also termed traditional, verification or cookbook) style employed at many post-secondary institutions (4, 6, 8). Each laboratory period is four hours long, where students are given enough time to complete the experiment and submit laboratory report sheets before the end of the session. At the start of each laboratory period, a pre-lab lecture is given either by the instructor to the 138 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

entire class (~80 students) or by the Teaching Assistants (TA) to smaller groups of generally less than 20 students. The pre-lab lectures highlight many of the important aspects of the experiment, such as reminders of safety considerations, explanation of experimental steps, information about the report sheets, and laboratory technique instructions/demonstrations among others. A drawback of the general pre-lab lecture is that students do not know which components of the lecture are the most important. If an instructor wants to highlight the importance of laboratory technique instructions, the pre-lab lecture is a poor way to stress this as students tend to assume that all material in the pre-lab lecture is of equal importance (9). There is also no mechanism to discern if students have received the message given in the pre-lab lecture. In regards to laboratory techniques development, there is no time to practice the skills highlighted in the pre-lab lecture (4). Also, the pre-lab lecture assumes that students are paying attention, and that they have come to the laboratory prepared to do the experiment (10). In our laboratory course, laboratory technique instructions were taught as part of the greater context of the course material. In order to stress to students the importance of learning proper laboratory techniques we introduced a Laboratory Techniques experiment into our curriculum (CHEM 126: General Chemistry Laboratory II). The introduction of the Laboratory Techniques experiment into the General Chemistry Laboratory II and not the General Chemistry Laboratory I (CHEM 121) was by design. As our students have diverse educational backgrounds, their laboratory experience varies with different skill sets learned and instructions given. By the end of CHEM 121, all students would have been exposed to the same level of instruction and skill development. As CHEM 121 is a pre-requisite of CHEM 126, we were able to use their CHEM 121 experience as a baseline in the development of the Laboratory Techniques experiment. With the new Laboratory Techniques experiment, students are presented with common laboratory techniques in one experiment at the beginning of the semester. Students are exposed to the same information previously taught in CHEM 126, but the method used to deliver the information has changed. As the new experiment is solely dedicated to proper laboratory technique skills development, students understand the importance of learning how to use the glassware and instrumentation correctly. This message is reinforced during the semester through spot-checks of their skills during the experiment, as well as the introduction of a practical exam, where their laboratory skills are tested (5). The effectiveness of new methods on student learning in laboratory courses is most commonly investigated through the use of student surveys (8, 10–14). Students are also examined through practical tests, written exams and through the development of student-generated videos where they explain and perform a specific skill (8, 11, 12). Although aspects of our assessment included a practical exam and qualitative observations, the overall effectiveness of our approach was determined by a direct analysis of the students’ laboratory skills and not their understanding of the theory or opinion on its effectiveness. We analyzed the experimental data to determine if there was an improvement in the precision of the data, which would occur if students understood and performed the laboratory techniques correctly. 139 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Laboratory Techniques Experiment: Plan 1 The initial version of the Laboratory Techniques experiment was broken into two parts. Part 1 consisted of instruction and practice time on six selected laboratory skills, while Part 2 required a review of the Laboratory Techniques manual and completion of a laboratory report sheet. Plan 1: Part 1

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

Laboratory techniques commonly employed in our first year experiments, as well as two theory based lessons comprised the topics of the Laboratory Techniques experiment. The topics were grouped as follows: • • • • • •

Pipettes Burette setup Titration Balances Volumetric glassware (Graduated cylinder and Volumetric flasks) PowerPoint presentations on: Lesson on Writing Skills (Purpose and Conclusion) and Calibration and Spectroscopy

The laboratory space consisted of 3x3 rows of benches with six students per bench (three per side). The room could be conveniently split into six stations with nine students assigned per station (54 students in total), hence the reason for the six topics. At each station students watched a demonstration given by the Teaching Assistant (TA) or instructor on the proper use of the glassware/instrument. Students were given the opportunity to practice the laboratory skills while the TA/instructor provided feedback. Students had to show the TA/instructor that they had mastered the skill by correctly performing to the TA/Instructor tasks such as “correctly filling the volumetric flask to the mark” or “dispensing 4.25 mL from the burette”. Incorrect steps were corrected and the student allowed to repeat the task. After 15 minutes, which was the time given for students to learn and practice the skills, they would rotate to the next station. The cycle was repeated until all students had gone through each station. At this point, students who had mastered all skills would proceed to Part 2 of the experiment, while those that did not were required to practice further until he/she could perform the technique correctly. Rarely did students require the extra help. Plan 1: Part 2 After the physical manipulation of the glassware/instrumentation in Part 1, students were asked to read a nine page Laboratory Techniques manual, which is a modified version of a Laboratory Techniques manual developed at BCIT. The manual covered all the topics of Part 1, except for the materials on the PowerPoint slides. A sample from the manual is provided in Figure 1 below. In addition to the Laboratory Techniques manual, a laboratory report sheet was provided to the students. The report sheet was due at the end of the laboratory 140 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

period. It was marked and returned in the following laboratory session. The laboratory report sheet consisted of 15 questions. The answers were either given in the demonstration at the technique stations or could be found in the Laboratory Techniques manual. Sample questions include: Why do you need to rinse (acclimatise) the pipette? Is it OK to pipette the liquid into the pipette bulb? Explain. Which finger do you use to control the liquid level in the volumetric pipette?

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

1. 2. 3.

Figure 1. A section of the Laboratory Techniques manual used in Part 2. Courtesy of Mrs. Rosamaria Fong, unpublished work.

141 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Reinforcement of the Message

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

The Laboratory Techniques experiment was performed in the second week of the semester. This allows students enough time to practice and apply their skill set when performing other experiments. The importance of using correct laboratory skills was reinforced throughout the semester. Students were reminded to use correct laboratory techniques in the pre-lab lectures. During the semester, TA spot-checks were done to assess students’ laboratory skills, and feedback was provided to them. A practical exam was also introduced into the course in which students’ laboratory skills were tested. Laboratory Technique Experiment: Plan 2 Although the students successfully developed correct laboratory technique skills through the method presented in Plan 1, the logistics of running Plan 1 presented limitations. Plan 1 was hindered in its ability to be applied to other situations. Our laboratory space conveniently allowed for six stations. With a maximum enrollment of 54, each group had a maximum size of nine students. With a different room size/configuration or increased enrollment capacity, the experiment would not work. Also with 54 students, 3 TAs were assigned to this course. For a laboratory section with a smaller enrollment there could be only 2 TAs. Plan 1 required a total of 5 TAs/instructor no matter the enrollment size, which meant volunteers had to be found to run the experiment. Running this experiment for students who were absent was also problematic as each station had to be set up individually. Due to the renovation of our building, we moved into a new space, which accommodated 80 students and 4 TAs in two rooms. Plan 1 would not work in this setup, therefore, it was modified and we developed Plan 2. Plan 2 removed the space and enrollment capacity limitations and allowed the experiment to be more adaptable to different situations. TA’s/Instructor demonstrations were replaced with PowerPoint presentations and an exercise sheet was added. Plan 2: Part 1 The laboratory techniques demonstration stations of Plan 1 were replaced with PowerPoint presentations, with the exception of the Manual Titration demonstration. This demonstration was shown to groups of 15-20 students. A test run of Plan 2: Part 1 was done with all demonstrations removed. It was found that both the Manual Titration demonstration and the PowerPoint presentation were required for the students to accurately learn the skill of a manual titration. Our students’ success rate in mastering the other skills did not diminish with the removal of the demonstrations and adoption of the PowerPoint presentations. All PowerPoint presentations were available to students on the laboratory computers (One computer per pair of students in the laboratory room), and they described the correct techniques to use/operate the glassware/instrument. The PowerPoint slides were accessible online after the laboratory session. A sample slide describing a Manual Titration is given in Figure 2. 142 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

Figure 2. Sample slide from the Manual Titration PowerPoint presentation.

Accompanying the PowerPoint presentations was the exercise sheet, which served two functions. Firstly, the exercise sheet was used to ensure that students read the PowerPoint presentation. Students had to answer on the exercise sheet questions that were embedded in the PowerPoint slides. Secondly, the exercise sheet was also used as a way to check whether students had mastered the laboratory techniques. Each exercise sheet included at least one task related to each technique, which students needed to perform for their TA. Each TA supervised a maximum of 20 students. If the students were successful in completing the task, the TA would sign the exercise sheet and award the student one point. In order to complete this portion of the experiment, students were required to obtain the TA’s signature for all laboratory techniques in order to prove that they had mastered all the laboratory skills. Students were given enough time to practice the technique until he/she mastered the skill, thus all students eventually receive full marks. A sample set of questions/tasks from the exercise sheet related to Burette and Manual Titration is shown in Figure 3 below.

Plan 2: Part 2 Once the students were able to perform all the techniques correctly they were given the same report sheet employed in Plan 1: Part 2. The same level of reinforcement was used throughout the semester. Students were reminded to use the correct laboratory techniques during their laboratory sessions. TA spot-checks were done to evaluate students’ laboratory skills, and provide feedback to the students. A practical exam was also given to test students’ laboratory skills. 143 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

Figure 3. Sample exercise sheet.

Plan 1 versus Plan 2 From a pedagogical point of view, both plans were found to be effective tools in teaching laboratory technique skills (this will be discussed further in the Results and Discussion section below). From the logistical point of view Plan 2 was superior over Plan 1. Plan 2 could easily adapt to new situations. The benefits of Plan 2 include the following: • • •

• •

•

The Laboratory Techniques experiment is no longer constricted by the laboratory space. More topics can be added to the Laboratory Techniques experiment. The Laboratory Techniques experiment is no longer restricted by the total number of students or the total number of TAs. There is no need to find volunteers to help with the experiment. As there is only one demonstration, “make-up” experiments are not difficult to set up. Each student receives the same information about the techniques. This was not guaranteed in Plan 1 with different TAs giving the same presentation in different laboratory sections (9). When reviewing the laboratory techniques, students have two resources to access: the PowerPoint presentations and the Laboratory Techniques manual.

144 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Two years after implementing Plan 2, we moved to our final laboratory space. Even with a very different layout of the laboratory space, we were able to continue to run the experiment successfully without modification to Plan 2.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

Results and Discussion (SFU) The effectiveness of this experiment was initially examined through qualitative observations. We found that students were more comfortable with the equipment, there were fewer problems with student data, and we were no longer asked by students “Why is my experimental data wrong when I did everything correctly?” We also performed a quantitative study of the effectiveness of this experiment. We theorized that since each student performs the same experiment, once students were taught correct laboratory techniques, the average values across all the laboratory sections should be very similar, and we should see an improvement in the precision of the student data when we compare student data prior to the implementation of the Laboratory Techniques experiment. (Table 1) Care was taken to minimize the external factors that could alter students’ results in this study. We chose to examine students’ results from Experiment 5, “Determination of an Equilibrium Constant” and Experiment 6, “The Solubility Product of Potassium Hydrogen Tartrate” from the CHEM 126 course laboratory manual. These experiments were chosen since both experiments did not undergo any revisions during the period of the study. The two experiments were also chosen as they made extensive use of the volumetric glassware such as burettes, volumetric flasks and pipettes, which requires proper laboratory techniques in order to achieve accurate measurements. For consistency, during the length of the study the same instructor also ran the experiments. We were able to collect students’ experimental results before and after the implementation of the Laboratory Techniques experiment, which allowed us to compare, quantitatively, the effect of the Laboratory Techniques experiment. Our students had a known laboratory technique skill set before completing this experiment, as almost all students would have taken the prerequisite laboratory course (CHEM 121) at SFU. The laboratory technique skill set developed in CHEM 121 did not change during the course of this study, therefore we are able to compare the different approaches of Plan 1 and 2, with the previous method used in CHEM 126 and make meaningful conclusions. Within each semester, the laboratory was run numerous times on different days, which we refer to as “lab sections”. For both experiments, students entered their results online in addition to their report sheet. This provided us with an electronic copy of the student values, which were used in our study. The value on the report sheet was used to ensure that the electronic data was entered correctly. For each lab section, we analyzed the student results to calculate an average value. We excluded all outliers that differed from the average value by more than three times the standard deviation. For semester 1 there were 4 outliers in the Experiment 5 data with 11 outliers for Experiment 6. For semesters 2-7 on average there was less than 1 outlier per semester for Experiment 5 and 4 outliers 145 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

per semeter for the Experiment 6. As shown in Table 1, for each semester we compared the standard deviation of the average value from each lab section. We limited the analysis of the student data to seven semesters as we did not have any data before semester 1. After semester 7 we made revisions to both experiments, which would have introduced too many variables into our study. From the data shown in Table 1, it is evident that Plan 1 and Plan 2 versions of the Laboratory Techniques experiment were successful in improving students’ laboratory skills through improved precision of the data. For Experiment 5, in all of the semesters after semester 1, we saw an increase in the precision of the student values as evident by the decrease in the standard deviation of the student results. The same trend is seen in Experiment 6, with the exception of semester 4. Although the standard deviation is still lower than that of semester 1, it is much higher than the rest of the values. During this semester we noticed that within the lab sections the student results would vary by more than 100%, but there was high precision within groups of students in the lab section. We suspected an issue with one or more sets of stock solution bottles. Although the erroneous results were reproducible, we were not able to definitively determine the cause. The information that can taken from the semester 4, Experiment 6 data, is that even with an additional source of error the precision of the data was still better than for Semester 1. This illustrates the effectiveness of the Laboratory Techniques experiment.

Table 1. Summary of student results from CHEM 126 (SFU) Experiment 5

Experiment 6

Method used

Semester

Number of Students

S.D.a

S.D. (x10-4)

“Old” way

1

262

0.142

1.91

Plan 1

2

96

0.061

0.22

3

157

0.035

0.61

4

296

0.050

--b

5

118

0.047

0.81

6

142

0.034

0.03

7

273

0.049

0.12

Plan 2

a

Standard deviations of the average values from each laboratory section within a semester. b This semester, experiment 6 had an unexplained source of error resulting in groups of students with results that were more than double of that of the other students in the same laboratory section. We suspect that there was an issue with one or more of the stock solutions bottles. The standard deviation of the data with the anomalies removed was 1.28x10-4, while for the whole data set it was 1.69x10-4.

146 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

After semester 7, instructional videos were introduced in these experiments. Videos on the techniques studied became available for students to view before their laboratory session. The effect of the videos on the precision of the students’ results has been difficult to assess since the experiments had undergone revisions, but a survey of the students found that they felt that the videos were an effective learning tool and their use should be expanded. The use of instructional videos as a teaching tool has been found to be an effective method to teach laboratory techniques through numerous studies (10, 12, 13, 15). The similar results of Plan 1 and 2 (Table 1) illustrated that the same message presented in two different methods can be used to impart laboratory technique skills to the student. The material presented in the Laboratory Techniques experiment was always part of the course. Previously, specific components of laboratory techniques were presented as part of the laboratory lecture for a particular experiment. A reason for the success of the Plan 1 and 2 versions of the Laboratory Techniques experiment was in grouping all the material into one dedicated experiment, laboratory skills became an important aspect of the course and students gave it greater attention.

British Columbia Institute of Technology (BCIT) BCIT is a polytechnic institute that offers a variety of programs where students are placed in “sets” (cohorts averaging 20 in size) to complete their studies together. The Chemical and Environmental Technology (CENV) program is a two year program (35 weeks per academic year) with two cohorts starting annually. Courses are taken over four consecutive school terms and students graduate with a diploma. Careers achieved by CENV graduates include analysts and research technologists in chemical laboratories. In order to prepare students, we focused on the following: • • •

Enhance students’ awareness and knowledge of chemical laboratory techniques required for employability. Provide greater opportunities for mastery of chemical laboratory techniques by practice and continuous feedback. Ensure that our evaluative component of chemical laboratory techniques is standardized, unbiased, and that the scoring methodology provides useful feedback to students for improvement.

The aim of the program is to teach practical laboratory skills in a cooperative learning environment. Our teaching methodology is designed to enhance learning through performance assessment and continual feedback (2). Here we describe the teaching methods used in the two first year chemistry courses in the CENV program, CHEM 1121 and CHEM 2204.

147 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

CHEM 1121 and CHEM 2204 Students entering the CENV program have varying chemistry backgrounds, ranging from students with university training to students with the minimum chemistry prerequisite of grade 12 high school chemistry. The students’ skills in a chemistry laboratory vary tremendously from year to year. As a result, CHEM 1121, a traditional chemistry course, is taken in the first term of the CENV program to provide students the necessary chemistry background (4, 6, 8). The course consists of 3 hours of lecture and a 3 hour laboratory period per week. During the laboratory periods, students are exposed to a wide variety of practical laboratory techniques as well this course serves as a foundation for career preparation. In the laboratory portion of CHEM 1121, students are introduced to basic volumetric glassware and weighing techniques. This included the correct use of volumetric flasks, pipettes, burettes, and different methods of weighing. Students were taught how to apply these techniques to perform manual titrations, gravimetric analysis, chromatography, and volumetric analysis. Since for many students this was the first formal introduction to chemical laboratory techniques, students were not graded on their laboratory techniques. However, to encourage students to achieve accurate results, they were graded on their analyses. The acceptable accuracy range of their results was less stringent than more advanced courses, such as CHEM 2204. A demonstration of the relevant techniques was given at the start of each laboratory period. Over the course of the term, students were given many opportunities to practice these laboratory techniques such that they become accustomed with the flow of the steps involved. Once students reached a comfortable level, a one-on-one peer review exercise was introduced where students performed laboratory techniques to a peer (16). The purpose of this exercise was to prepare students for a job interview, where one could be asked to demonstrate a laboratory technique to a certain level of competency. In order to quantify the level of competency and standardize an unbiased evaluative component, we introduced laboratory technique assessment scoring rubrics for each laboratory technique. For example, Figure 4 shows a sample of the laboratory technique rubric used to measure the competency of using a burette during a manual titration. Scoring rubrics helped students determine specifically what was required of them and these tools also help new instructors provide consistent demonstrations across multiple course sections (17). The peer review exercise allowed students to learn and evaluate chemical laboratory techniques. Students, as reviewers, gained experience in assessing the laboratory techniques demonstrated by their peers. Using the laboratory technique rubric, students evaluated each other and practiced providing verbal constructive feedback. For the student being reviewed, they gained experience in maintaining their composure while demonstrating their skills. This exercise helped prepare students for the next chemistry course, CHEM 2204, where they were evaluated via formal laboratory technique assessments by an instructor who used the same laboratory technique rubric. 148 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

Figure 4. Laboratory Assessment Rubric for Technique 1 – Use of Burette CHEM 2204 is the second term course in the CENV program. It is a 20-week course for 3 hours per week with the following laboratory components: • • • • •

Techniques and Practice (T&P) Traditional chemistry laboratories Data manipulation and Graphing Practical assessments (PA) Final laboratory exams

An open course website was set up for students to access course information (18). The reference text for this course is the “Chemical Technicians Ready Reference Handbook” (19). Techniques and Practice (T&P) We introduced a set of T&P laboratories, where students were given additional opportunities to practice laboratory techniques. These T&P laboratories alternated with traditional chemistry laboratories (see below), where the experiments are chosen with procedures relying largely on using skills learned. During the T&P laboratories, students were expected to continue peer reviewing each other using the technique rubric given. No mark was awarded for this component of the course. 149 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Traditional Chemistry Laboratories

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

There were five traditional chemistry laboratories where students carried out laboratory analyses, perform calculations, and reported the results of their analyses. These laboratories were based on titration methods, gravimetric analysis, weighing, colorimetric analysis and extraction. The reported results were graded out of a score of ten. Depending on the laboratory, a maximum of six marks were awarded for accuracy and precision, and the other marks were distributed for correct calculations and observing significant figure rules in their calculations and results. Data Manipulation and Graphing Another employable skill to emphasize for students is data manipulation of experimental results (20, 21). There were five graphs in this component of the course. Demonstrations were given in class on the use of Microsoft Excel to manipulate data on a spreadsheet, analyze and report statistical errors, and produce graphs with proper labelling. To expand on the benefit of cooperative learning, 20% of the graphing mark was awarded for the ability of students to review another student’s graph (22). Students printed a copy of their graph and exchanged graphs with another student for peer review. Using the peer reviewer’s feedback, students made changes to their original graph and compiled a set of the following three graphs to the instructor: • • •

Graph 1 is Student A’s original graph. Graph 2 is Student A’s graph that was reviewed by Student B. Student B’s comments were annotated on this graph. Graph 3 is Student A’s final corrected graph.

The instructor marked Graph 2 and Graph 3 for a total score of 10 using the following guideline: • •

Student A is graded on his/her ability to peer review Student B’s graph. Student A’s comment will appear on Student B’s Graph 2. (2 marks) Student A’s Graph 3 is graded on the completeness of the graph. (8 marks)

A sample submission of the graphs from Student A is provided in the Results and Discussion section below (Figure 5). Practical Assessments (PA) There were two 15-minute one-on-one Practical Assessments (PA) in the course: PA 1 in Week 8 and PA 2 in Week 15. Table 2 summarizes the laboratory techniques that students were responsible to learn to demonstrate their knowledge. Because of the 15-minute time constraint, students could only be effectively assessed on two techniques. On the day of the assessment, students were randomly assigned two techniques. During their assessment, the instructor asked students to 150 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

explain keywords specific to the technique that was being assessed. Students were expected to communicate calmly and articulate scientifically sound responses. In the time given, students were required to complete a minimum number of steps for each technique. The instructor observed the student and graded the student with the same laboratory technique rubric that they have used in their peer review exercise.

Table 2. Summary of the techniques assessed in week 8 and week 15

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

Practical Assessments – Techniques assessed Week 8

Use of a Pipette, Weighing, Use of a Bottle-Top Dispenser, Gravity Filtration, Rotary Evaporator

Week 15

Use of a Burette, Use of a Volumetric Flask, Preparation of Standard Solution, Titration

The following week, the instructor conducted a one-on-one formal discussion about each student’s proficiency in carrying out the techniques assessed. This usually required between 3 to 5 minutes, where the instructor gave constructive feedback to communicate the following: • • •

What the student has mastered, and the extent of that mastery on the two techniques that were assessed. What the student needs to improve on to perfect the techniques. What the student needs to improve on in terms of their communication skills.

Final Laboratory Exams The goal of the final laboratory exam was to help students prepare for their future career by providing situations for authentic learning (23). In the final two weeks of the course, students carried out two laboratory exams. Table 3 summarizes the grading scheme of the laboratory exams. These exams were completed individually, with the entire class carrying out the laboratory exam at the same time. This is a more realistic situation found in a workplace, where the attention is not usually focused one-on-one. The instructor played the role of a laboratory supervisor who monitors a group of laboratory analysts as they carry out an analysis at their own bench space. Rules were set for the usage of the equipment. For example, since there was a limited number of analytical balance, students were restricted to weighing one sample at a time. The instructor noted when students use incorrect techniques. Each incorrect technique was a demerit of 0.5 marks, up to 7 marks. 151 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

Table 3. Laboratory exam grading scheme Laboratory Techniques

7 marks

Accuracy of Results

6 marks

Calculations

4 marks

Organization and the ability to follow a procedure

3 marks

Total

20 marks

CHEM 2204 was designed to help the student focus on the finer details of the laboratory techniques that may be missed while observing a laboratory demonstration. In addition to the techniques previously discussed by SFU, the full version of the BCIT Laboratory Techniques manual included the following procedures: gravity filtration, use of the rotary evaporator, calibration and use of the bottle-top dispenser.

Results and Discussion (BCIT) Overall, students who conscientiously practiced their laboratory techniques while being reviewed by a classmate using the scoring rubrics decreased their anxiety of making mistakes. Similar to SFU, students demonstrated more motor proficiency than the first set of attempts with the techniques and confidence when carrying out the laboratory techniques. We have not carried out a detailed numerical analysis of student performances because the students’ backgrounds vary significantly from year to year (i.e. we do not have a controlled set of students with the same chemistry background). Due to time limits of the one-on-one practical assessment, each student also cannot be assessed on all the techniques practiced in the T&P weeks. We remind students during the term that all laboratory techniques will be assessed in the final laboratory exams. Table 4 shows the typical results for a recent group of students for the Practical Assessments (PA) and final laboratory exams. The scoring rubric from the PA was reviewed with each student in a one-on-one discussion. Students have used the analogy of “the road test for a driver’s license” when they carried out the practical assessments. There are always two to three students who score below 50% on the first practical assessment due to anxiety. We advise those students to modify their preparation specific to their performance and incorporate study skills, group work or student services that reduce anxiety. Graduated CENV students commented that the practical assessments helped them prepare for their second year industry practicum and job interviews which required a demonstration of a laboratory technique (e.g. laboratory technician).

152 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Table 4. Results for Practical Assessments and Final Laboratory Exams Average % a

Standard Deviation

Range of Marks

74

3

22-97%

76

3

50-95%

Final Laboratory Exam I

73

3

51-90%

Final Laboratory Exam II

71

2

56-90%

Evaluative Component Practical Assessment Week 8 b Practical Assessment Week 15

c

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

a

Results from 32 students. PDF files of all the scoring rubrics can be found on the course website (18). b Two of the following techniques assessed: use of a pipette, weighing, use of a bottle-top dispenser, gravity filtration, and rotary evaporator c Two of the following techniques assessed: use of a burette, use of a volumetric flask, preparation of a standard solution and titration

In the past 6 years that the CENV program has been offered, our feedback indicates that students are more aware of the following expectations: • • • •

Competency in the laboratory techniques students need to master at the end of the first year of the CENV program. Competency in preparing graphs to display a data set. Competency to communicate cohesively and logically using appropriate chemically correct keywords. Competency to work collaboratively and provide constructive criticism.

Our training better prepares students for the second year of their program as well as employability after graduation. For the collaborative learning environment featured in the courses, students see the advantages of receiving immediate feedback and working collegially with their instructor and peers to improve their own learning (22). Students view the grading system to be fair because they get a second chance to check for improvement. The peer review of graphing assignments promoted a cooperative learning environment, team work, and communication among students. Figure 5 is a typical set of graphs that students submitted for grading. Without the peer review process, Student A would have submitted Graph 1 for grading, possibly missing many important aspects of the graph. However, Graph 1 was peer reviewed by Student B, who clearly marked up the incorrect points in Graph 2. Student A was, therefore, able to correct his/her mistakes and submit Graph 3 for grading. This was a discipline-based process which allowed students to experience the benefit of working collaboratively. It also provided students with the opportunity to implement corrections and submit their best work.

153 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

Figure 5. Graphing and Peer Review submission.

154 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

Conclusion In order to effectively teach students proper laboratory technique effort must be made to make the subject important. This can be accomplished in the form of a dedicated Laboratory Techniques experiment or through the use of technique specific assessment rubrics that carry over numerous courses with different levels of evaluators (i.e. their peers and instructor). The goal of developing sound laboratory technique and transferable skills in our students was modified to meet the objectives of the programs at SFU and BCIT. At SFU, both iterations of the Laboratory Techniques experiment improved the laboratory technique skills of the students, as evaluated through qualitative observations and the quantitative results shown in Table 1. Through improved laboratory skills, students were better prepared to perform experiments, obtain results that match theoretical predications and become more comfortable in the laboratory setting. At BCIT, the “learn, practice, do, review” cycle built into the first year chemistry courses has shown to instill confidence and skills in the students in the CENV program. Although students enter the program with varying skill levels, the first year chemistry courses help to set a baseline of laboratory skills expectations, develop and refine students’ laboratory techniques. The courses are developed to give the students transferable job skills. Based on the BCIT Student Outcomes Reports for Certificate and Diploma Graduates, 79% of the graduates reported the program as useful in helping them find employment (24).

Acknowledgments The laboratory technique videos were supported by a Teaching and Learning Development Grants from the Institute for the Study of Teaching and Learning in the Disciplines (ISTLD) and the Teaching and Learning Centre (TLC) at Simon Fraser University (25). Debbie Owen (SFU) is acknowledged for her assistance in the development of the Laboratory Techniques experiment. We also would like to thank all students who “tested” our ideas in the development of these learning tools. We are grateful to SFU and BCIT for their support and funding.

References 1. 2. 3.

4. 5.

Faraday, M. Chemical Manipulations; Wiley: New York, 1827. DeMeo, S. Teaching chemical technique: A review of the literature. J. Chem. Educ. 2001, 78, 373–379, DOI: 10.102/ed078p373. Burgard, D. A. An introduction to the general chemistry laboratory that makes a lasting impression concerning laboratory safety. Chem. Educator. 2005, 10, 427–429, DOI: 10.1333/s00897050966a. Domin, D. S. A review of laboratory instruction styles. J. Chem. Educ. 1999, 76, 543–547, DOI: 10.1021/ed076p543. Hofstein, A. The laboratory in chemistry education: Thirty years of experience with developments, implementation and research. Chem. Educ. Res. Pact. 2004, 5, 247–264, DOI: 10.1039/b4rp90027h. 155

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

6.

7.

8.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

9.

10.

11.

12.

13.

14.

15.

16.

17.

Cacciatore, K. L.; Sevian, H. Incrementally approaching an inquiry lab curriculum: Can changing a single laboratory experiment improve student performance in general chemistry? J. Chem. Educ. 2009, 86, 498–505, DOI: 10.1021/ed086p498. Galloway, K. R.; Brats, S. L. Measuring meaningful learning in the undergraduate chemistry laboratory: A national, cross-sectional study. J. Chem. Educ. 2015, 92, 2006–2018, DOI: 10.1021/acs.jchemed.5b00538. Dunlap, N.; Martin, L. J. In Advances in Teaching Organic Chemistry; Duffy-Matzner, J. L., Pacheco, K. A. O., Eds.; ACS Symposium Series 1108; American Chemical Society: Washington, DC, 2012, pp 1−11, DOI: 10.1021/bk-2012-1108.ch001. Majerle, R. S.; Utecht, R. E.; Guetzloff, C. J. A different approach to the traditional chemistry lab experience. J. Chem. Educ. 1995, 72, 718–719, DOI: 10.1021/ed072p718. Key, J.; Paskevicius, M. Investigation of video tutorial effectiveness and student use for general chemistry laboratories. J. App. Learn. Technol. 2016, 5, 14–21. Towns, M.; Harwood, C. J.; Robertshaw, M. B.; Fish, J.; O’Shea, K. The digital pipetting badge: A method to improve student hands-on laboratory skills. J. Chem. Educ. 2015, 92, 2038–2044, DOI: 10.1021/acs.jchemed.5b00464. Jordan, J. T.; Box, M. C.; Eguren, K. E.; Parker, T. A.; Saraldi-Gallardo, V. M.; Wolfe, M. I.; Gallardo-Williams, M. T. Effectiveness of studentgenerated video as a teaching tool for an instrumental technique in the organic chemistry laboratory. J. Chem. Educ. 2016, 96, 141–145, DOI: 10.1021/ acs.jchemed.5b00354. Popova, M.; Bretz, S. L.; Hartley, C. S. Visualizing molecular chirality in the organic chemistry laboratory using cholesteric liquid crystals. J. Chem. Educ. 2016, 93, 1096–1099, DOI: 10.1021/acs.jchemed.5b00704. Mathew, J. M.; Grove, N.; Bretz, S. L. Online data collection and database development for survey research in chemistry education. Chem. Educ. 2008, 13, 190–194, DOI: 10.1333/s00897982133a. Canal, J. P; Hanlan, L.; Key, J.; Laveiri, S.; Paskevicius, M.; Sharma, D. Chemistry Laboratory Videos: Perspectives on Design, Production, and Student Usage. In Technology and Assessment Strategies for Improving Student Learning in Chemistry; Schultz, M., Schmid, S., Holme, T.; , Eds.; ACS Symposium Series 1235; American Chemical Society: Washington, DC, 2016; Chapter 9. Ponrello, J. K. Enhancing the skill-building phase of introductory organic chemistry lab through a reflective peer review structure. J. Chem. Educ. 2016, 93, 262–269, DOI: 10.1021/acs.jchemed.5b00655. Chen, H.; She, J.; Chou, C.; Tsai, Y.; Chiu, M. Development and application of a scoring rubric for evaluating students’ experimental skills in organic chemistry: An instructional guide for teaching assistants. J. Chem. Educ. 2013, 90, 1296–1302, DOI: 10.1021/ed101111g.

156 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch008

18. Fong, R. BCIT CHEM 2204. nobel.scas.bcit.ca/courses/wpmu/chem2204/ laboratory-2/ (accessed July 20, 2016). Note: Scoring rubric PDFs are linked in the Practical Assessment Section. 19. Shugar, G. J., Ballinger, J. T. Chemical Technicians’ Ready Reference Handbook, 4th ed.; McGraw-Hill: New York, 1996. 20. Rubin, S. J.; Abrams, B. Teaching fundamental skills in Microsoft Excel to first year students in quantitative analysis. J. Chem. Educ. 2015, 92, 1840–1845, DOI: 10.1021/acs.jchemed.5b00122. 21. Ashraf, S. S; Marzouk, S. A. M.; Shehadi, I. A.; Murphy, B. M. An integrated professional and transferable skills course for undergraduate chemistry students. J. Chem. Educ. 2011, 88, 44–48, DOI: 10.1021/ed100275y. 22. Berry, D. B; Fawkes, K. L. Constructing the components of a lab report using peer review. J. Chem. Educ. 2010, 87, 57–61, DOI: 10.1021/ed8000107. 23. Lombardi, M. M. Authentic learning for the 21st century: An Overview. Educause Learning Imitative Website. https://net.educause.edu/ir/ library/ pdf/eli3009.pdf (accessed March 1, 2016) 24. BCIT Student Outcomes Reports for Certificate and Diploma Graduates 2012 Edition website. http://www.bcit.ca/files/ir/ pdf/gradoutcomes_ dacso _2 page_2012.pdf (accessed March 22, 2016); pp 288−289. Note: The Chemical Sciences Program was renamed the Chemical and Environmental Technology Program after a curriculum change in 2010. 25. SFU Teaching and Learning Development Grant Home page. http:// www.sfu.ca/tlgrants.html (accessed March 25, 2016).

157 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Chapter 9

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

Chemistry Laboratory Videos: Perspectives on Design, Production, and Student Usage John P. Canal,*,1 Lee Hanlan,1 Jessie Key,*,2 Sophie Lavieri,1 Michael Paskevicius,3 and Dev Sharma1 1Department

of Chemistry, Simon Fraser University, 8888 University Drive, Burnaby, British Columbia, V5A 1S6 Canada 2Department of Chemistry, Vancouver Island University, 900 Fifth Street, Nanaimo, British Columbia, V9R 5S5 Canada 3Centre for Innovation and Excellence in Learning (CIEL), Vancouver Island University, 900 Fifth Street, Nanaimo, British Columbia, V9R 5S5 Canada *E-mails: [email protected] (J.P.C); [email protected] (J.K.)

The effectiveness of instructional videos as a teaching tool in the chemistry laboratory curricula at both Simon Fraser University (SFU) and Vancouver Island University (VIU) is examined. Five categories of videos used in first, second and third year laboratory courses were developed, either in-house (by faculty) or with the assistance of visual media professionals. Short student feedback surveys from both institutions indicate that students find the videos to be an effective tool in their education. Most students felt they were better prepared and more confident about their experiments after watching the videos.

© 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

Introduction Laboratory experiences are vital components of a post-secondary chemistry education. Laboratory work allows for the development of observational, problem-solving, inferential and technical (manipulative) skills (1, 2). Furthermore, the reinforcement and exploration of chemical theory through physical manipulations is a fundamental aspect of most chemical laboratory courses. The laboratory setting is a means to present material in a different format from lecture and expose the students to new skills. Information presented to the students in the laboratory setting generally consists of a pre-laboratory exercise/assignment, a laboratory lecture and then the completion of an experimental procedure using instruction written in a laboratory manual. Efforts have been made to maximize students’ learning potential by introducing new education tools and adopting new technology (3–7). In the chemistry departments of both Simon Fraser University (SFU) and Vancouver Island University (VIU), video technology was independently introduced into our laboratories through the development of instructional videos, that can be classified in five broad categories: Laboratory Techniques, Laboratory Safety, Experimental Procedure, Instrumentation, Theory and Calculations. The videos are predominantly used to enhance the pedagogical strategies currently used in our courses. These videos were created with the intent of improving students’ laboratory learning experience through the introduction of educational technology. Although the projects at SFU and VIU were independently conducted, a comparison of the collected data showed common results. The use of educational technology in our laboratories should serve a distinct pedagogical goal (8). Educational goals often drive pedagogical approaches including those which make use of educational technology (9). As educational technology is not employed in a standardized way in higher education, the challenge is in raising awareness of how educational technology might facilitate the achievement of educational goals (10). For an educational technology intervention to move from experiment to common practice, an instructor must perceive it adds some value to their work (11). Educational technologies have the potential to transform the ways in which educational experiences are conducted. An emerging model for using educational technology called the “flipped” or “inverted” classroom suggests that content transmission happens outside of the classroom. In this model, students watch video recorded lectures online and then come to class to discuss and practice the content further. This model promotes a more active learning environment in the classroom and has been demonstrated to result in better student performance (12). Despite the promise, the use of educational technologies remains largely unexploited. A recent report commissioned by the Bill and Melinda Gates Foundation sought to explore instructors’ experience working with educational technology. Most instructors in the study indicated a familiarity with contemporary educational technologies including an awareness of some of the potential benefits but fewer had adopted educational technologies in their teaching practice (9). Notable barriers to experimentation include a feeling that innovative teaching goes unrewarded. Instructors also reported that the time and 160 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

effort involved to use educational technology was not incentivized. Furthermore, some instructors may feel uncomfortable learning or using new educational technologies. Stable and ongoing funding to directly support the development of educational technologies is a highly unlikely prospect. Thus, other motivating factors should be explored to entice instructors into developing stand-alone learning resources. We propose three potential benefits resulting from the creation of video resources to support the chemistry curriculum. The videos are a reusable digital resource, they offer an effective pedagogical approach to present safety and relevant content to students, and provide a means to collect data to inform our teaching practice. Reusable Digital Resources Visuals developed in a class or laboratory setting are subject to variation each time when they are produced on a chalk or whiteboard. Furthermore, they are wiped away as the board is cleaned and generally not captured for further use or access. While the development of digital multimedia visuals can take a significant amount of time, these visuals can be shared and replicated easily for use by many students on demand. Being electronic resources they are also easy to edit, manipulate or customize as needed. Over time a library of these visual teaching materials can be built up, providing an ongoing resource for instructors and students. While there is a compromise between the time invested creating an online video resource and how many students are impacted by that resource, once the resource has been created it can be reused, adapted and shared effortlessly. This saves preparation time in the future, and the resource is available for use by colleagues or future instructors. Effective Pedagogical Approach As an additional tool used to educate students, instructional videos enhance core methods of instruction. As students learn at different rates and with different preferred methods, these videos provide another means to engage students. Students are able to review the material on their own time, with the option of replaying, rewinding and controlling the flow of information. Rather than delivering this content in the laboratory, face-to-face time can then be used to allow students to question and practice what they learned through the videos. Students can be given time to consider, practice and reflect on the material on their own time and subsequently bring more significant questions to the laboratory for discussion. Valuable laboratory time can then be used to discuss problems or issues with the content, allow hands-on usage of instruments and equipment, and practice technique or skills collaboratively. In this way students are given significant ownership of their own learning, rather than trying to keep up with the content delivery in the laboratory. Furthermore, control is granted to students who feel they are doing well with the material and can choose not to use the videos. Building the transmission of content into educational technology video resources, 161 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

which are reviewed outside of the classroom, may also provide opportunities for more student discussions within the laboratory.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

Data Informed Practice Conventionally, instructors would not have the opportunity to track students and their access to resources, or lack thereof, which occurs outside of the classroom or laboratory. Consequently, there would be no way for an instructor to know definitively if a student had accessed a required reading prior to a laboratory period. Students who access instructional videos generate activity data as they move through online systems. Web analytics can be used to track the access of resources and videos watched. If a user is logged in while viewing a video, one can view their individual progression as they interact with resources and assessments. This data can be used to speculate which concepts are proving most difficult for students based on user activity. Additionally, instructors may make use of web analytics to track how students are using materials and plan interventions or changes to the lessons. Web analytics data can provide a view to observe which students have accessed videos online. These analytics may expose students who are not engaging with the resources and provide opportunities for intervention and support.

Development of Laboratory Videos Within the chemistry departments of both Simon Fraser University (SFU) and Vancouver Island University (VIU) faculty members saw the opportunity to create instructional videos to assist student learning. The approach taken to produce these videos, the challenges encountered and the pedagogical aspects of the videos will be investigated. The development and production of instructional videos follows a basic structure that applies to both in-house produced videos (VIU) and video created by visual media professionals (SFU) (Figure 1). The first step in this process is to identify which learning outcomes or laboratory topics would benefit from a video tutorial. Reflection on the learning outcomes and expected literacies of the laboratory course is an important preliminary step towards the production of quality laboratory videos. Instructors should take into consideration all aspects of the laboratory experience to ensure that all desired outcomes are addressed while avoiding duplication or unnecessary overemphasis of certain outcomes/skills. This includes information spanning pre-laboratory (before the laboratory period begins) preparation, during the laboratory and post-laboratory (after the laboratory period has ended). Existing student knowledge and common misconceptions should be examined to help guide the design and flow of the video material. After a topic has been chosen, the writing of a script helps ensure a clear, concise message is presented in a logical order. The process of filming and recording begins by setting up the required equipment or glassware, followed by the actual recording of techniques or screen capture. Video recording may be performed simultaneously 162 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

or separately from audio recording. It is critical that both processes are done at the highest possible quality to extend the shelf-life and appeal of the videos produced. Editing and merging of the recorded video and audio should be performed to further enhance the production quality of the videos. Finally, the finished videos should be published to a reliable delivery system which all students can access, regardless of the device or operating system they chose to use.

Figure 1. Flow chart used in the development of instructional videos. Vancouver Island University (VIU) The development of the VIU video series was supported by a modest “Learning Innovation and Enhancement in Teaching (LITE)” grant from the Vancouver Island University Centre for Innovation and Excellence in Learning. All aspects of the instructional video development chart (Figure 1) were carried 163 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

out by primarily one faculty member with the assistance of laboratory technicians and students in the filming of live-action demonstrations. Live action was recorded with a high-definition flash memory camcorder and audio voiceovers were recorded using a high definition USB microphone. Video and audio editing were performed using the freeware software Active Presenter (13) and Audacity (14). Some additional editing was performed using Windows Live Movie Marker (Version 2011) (15) and Adobe Premiere Elements 9 (16). Videos were published using VIU’s Kaltura based video media repository (VIUtube) and embedded into a course’s learning management system Desire2Learn (17).

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

Simon Fraser University (SFU) The development of the SFU video series was supported by four “Teaching and Learning Development Grants” from SFU. The first four steps of the instructional video development chart (Figure 1) were carried out by our lecturers, specifically those listed as co-authors of this manuscript. We developed the ideas for the videos, worked together to finalize the script and “shots” required and had input on the final edits of the videos. With the support of the “Teaching and Learning Development Grants”, we were able to contract out to a private video production company the filming, audio/voice over and final editing of the videos, though we worked with them on each step of the process. The video production company also developed two videos using whiteboard animation. The final product was posted by SFU on an internal YouTube channel and the link was either provided to the students through email or embedded into LONCAPA (18) or Canvas (19), which are course learning management systems used by these courses. Categories of Laboratory Videos Developed A benefit of instructional videos is the versatility in their applications. The videos developed at SFU and VIU can be grouped into five general categories, as listed below with the complete list of videos shown in Table 1: 1. 2. 3. 4. 5.

Laboratory Techniques Laboratory Safety Experimental Procedures Instrumentation Theory and Calculations

In general the instructional videos were used as an additional or supplemental tool in presenting the material to the students. The students were instructed to view the videos prior to commencement of the experiment where the material would be implemented. The videos were used in conjunction with other instructional methods, such as pre-lab lectures and reference material written in the laboratory manual. All videos developed at VIU were used as supplements to the existing laboratory manuals and pre-laboratory instructional briefings for the 164 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

courses: Chemistry Fundamentals I and II (CHEM 140 and 141/142) Engineering Chemistry (CHEM 150) and Organic Chemistry I and II (CHEM 231 and 232).

Table 1. Instructional videos developed at SFU and VIU Disposable pipettesa

How to use a burettea

Separation by flotationa

Weighinga

Pipettinga

Titrationa,b

IR KBr methodb

Vacuum filtrationb

IR nujol mull methodb

IR thin film methodb

Melting pointsb

Volumetric glasswareb

Performing a distillationb

Gas chromatography preparation and useb

Column chromatography demonstrationb

One solvent recrystallizationb

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

Laboratory Technique

Preparing an NMR sampleb

Polarimetryb

Fume hooda

Glove boxa

Schlenk linea

Gas cylindera

Lab orientation and safety Vol. 1b

Lab orientation and safety Vol. 2b

Globally Harmonized System (GHS)b

Experimental Procedure

Chromium acetate hydratea

Collection of H2 gas over waterb

Instrumentation

Agilent 6890 GC-MSa

Using a viscometerb

STA-6000 (thermal analysis)a

Flame atomic spectrometera

UV-Vis (Spec 20) use/theoryb

Laboratory Safety

pH meter calibration and useb Theory and Calculations

a

Concentration cellsa

Dilutionsa

Unit conversionb

Precision and accuracyb

Distillation theoryb

Experimental errorb

Organic lab calculationsb

TLC theoryb

Stoichiometryb

Significant figuresb

Gas chromatography theoryb

Column chromatography theoryb

Plotting a calibration curve in Excelb

Processing an IR spectrum using Omnic softwareb

Liquid-liquid extraction theory and demonstrationb

Interpreting an IRb

Recrystallization theoryb

SFU.

b

VIU.

At SFU the videos were employed in the following courses: General Chemistry and Laboratory I (CHEM 121), General Chemistry Laboratory II (CHEM 126), Introduction to Analytical Chemistry (CHEM 215), Inorganic Chemistry Laboratory (CHEM 236) and Advanced Inorganic Chemistry Laboratory (CHEM 336). 165 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

Except for the “Separation by Flotation” video, the Laboratory Techniques videos developed at SFU are used in a dedicated Laboratory Technique experiment developed for CHEM 126. The experiment makes use of the videos in addition to PowerPoint slides, hands on practice, a laboratory technique manual, as well as instructor demonstrations. This approach to teaching laboratory techniques, via a Laboratory Techniques experiment has proven quite effective (20). The Laboratory Technique experiment allowed us to probe the method of instruction preferred by students and this is discussed in the Evidence of Success/ Student Usage section below. The Instrumentation videos developed at SFU are different as they were not used as an additional tool to enhance other methods of instructions, rather to replace one. These videos were developed for a second year analytical chemistry laboratory course (SFU: CHEM 215). The approach taken in this course was to have the videos replace the instructions on the use of the instruments written in the course laboratory manual. Prior to the experiment, students are required to view a video and write out the experimental procedure on the proper use of the instrument. It has been found, that with this approach students arrive to the laboratory more prepared to complete the experiment. The student feedback on this use of the Instrumentation based videos has also been positive and will be discussed below. Evidence of Success/Student Usage While video technology has been examined as an instructional resource in chemistry laboratories since the 1970’s there have not been many studies that examine the students’ consumption and activity data (7, 21–23). Preliminary studies have compared videotaped chemistry laboratory demonstration effectiveness to written instruction with respect to cognitive understanding and manipulative skill ability. Kempa and Palmer found that students who were taught manipulative laboratory skills by recorded video demonstration on a television set showed superior performance compared to students who received only written instruction (22). This study demonstrates students may benefit from visualization of laboratory techniques, but does this translate to student success in a modern context? Students today may, or may not, choose to consume available streaming video resources using a variety of devices (computers, mobile devices etc.) for a variety of purposes - before, during or after their laboratory periods. A study by Teo et al. examined the use of video demonstrations of laboratory procedures for Year 1 Inorganic and Year 2 Organic Chemistry courses at Nanyang Technological University, Singapore (24). The authors used web-browser and smartphone application delivery systems to replace their traditional pre-laboratory briefings with demonstration videos followed by pre- and post-laboratory discussions during the laboratory period. The effectiveness of this method was examined qualitatively by performing interviews with students and observing their performance in lab. The authors suggest that undergraduate students developed a better understanding of the theory of the laboratory steps, and were more confident performing procedures, resulting in faster progress through the laboratory objectives (24). 166 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

A recent study at VIU was conducted to assess laboratory video tutorial student usage quantitatively in general chemistry labs, to assess if laboratory video tutorials based on modern technology aid student learning (17). A series of live-action and screencast videos were created to supplement existing methods (lab manual and in-laboratory instruction) of conveying safety, theory and techniques in the first year chemistry courses Chemistry Fundamentals I (CHEM 140) and Engineering Chemistry (CHEM 150). The authors then used web analytic tracking functions including Google Analytics and video streaming server analytics data to obtain quantitative usage data, while surveying student and faculty for qualitative feedback. With a student sample size that began at 229 and decreased to 184 by the end of the semester, individual videos were played between 33 to 122 times during the semester, with a mean value of 63 plays. Of the total number of plays, the mean percentage which were watched to completion (playthrough ratio) for all videos was 68.6%.

Figure 2. Number of video tutorial pageviews per week during the Fall 2013 semester for CHEM 140/150 (VIU). (* No lab session, # No applicable video for this lab). Server data enabled the visualization of student viewing habits throughout the semester, as represented by the plot of video resource page views per week (Figure 2). This plot supports that students used the video resources to prepare for assessment with a noticeable spike in pageviews during the week of laboratory examinations (week 13). Student survey results from 78 voluntary respondents in CHEM 140/150 during the Fall 2013 semester, indicated students chose to access laboratory video tutorials throughout the course (66%). In addition, the majority of responding students who watched the video tutorials felt they gained a deeper understanding of the laboratory materials (83%) and had more self-confidence 167 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

entering their weekly labs (88%) (Table 2). This study suggests that responding students find value in using video tutorials to prepare themselves for weekly laboratory experiments and to review for examinations. It further supports the qualitative findings of Teo et al. (24)

Table 2. Summary of student survey for CHEM 140/150 (VIU)a

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

1. How helpful were the video tutorials? Very helpful

Slightly helpful

Not at all helpful

Did not watch them

51%

15%

0%

33%

2. Did you experience any technical issues while viewing the videos? Yes always

Yes sometimes

No not at all

Only on certain devices

0%

12%

81%

8%

3. What are the reasons you watched the video tutorials? (Check all that apply) To prepare before lab

Help answer post-lab questions

Prepare for lab quiz/exam

Other

94%

31%

50%

12%

4. People use video tutorials in many ways. Which is the single best description of how you typically used the video tutorials? Watched entire video: start to finish

Watched sections looking for information

Browsed around

Re-watched segments

Went to specific points to review

77%

15%

2%

4%

2%

5. Rate your level of agreement with the following statement: I have a deeper understanding of the lab material because of the instructional videos. Strongly agree

Agree

Neutral

Disagree

Strongly Disagree

12%

71%

13%

2%

2%

6. Did you feel more confident in the lab having watched the instructional videos before attending lab? Continued on next page.

168 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Table 2. (Continued). Summary of student survey for CHEM 140/150 (VIU)a Strongly agree

Agree

Neutral

Disagree

Strongly Disagree

27%

61%

12%

0%

0%

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

7. If you did not watch certain tutorial videos, what are the reasons? (Check all that apply)

a

Confident with material already

Forgot

Not enough time

Technical problems

Other

36%

48%

48%

4%

4%

Data taken from Reference (17).

The effectiveness of the Laboratory Techniques videos used in CHEM 126 (SFU) was investigated through a voluntary online student feedback survey (Table 3) of the students enrolled in the course in the spring 2016 semester. Of the 195 students enrolled in the course, 56 completed the survey.

Table 3. Summary of student survey for CHEM 126 (SFU) 1. Did you watch any of the instructional videos? Yes

No

98%

1%

2. How helpful were the instructional videos? Very helpful

Somewhat helpful

No at all helpful

41%

57%

2%

3. The technique of "Manual Titrations" was presented to you in four different ways. Please list the following methods of instruction in order of the most effective (1) to the least effective (4). 1

2

3

4

PowerPoint slides

11%

34%

34%

21%

Instructor demos

46%

18%

18%

18%

Reading the manual

7%

27%

34%

32%

Watching the videos

29%

30%

21%

20%

4. Rate your level of agreement with the following statement: I have a deeper understanding of the lab material because of the instructional videos. Continued on next page.

169 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Table 3. (Continued). Summary of student survey for CHEM 126 (SFU) Strongly agree

Agree

Neutral

Disagree

Strongly Disagree

13%

70%

16%

1%

0%

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

5. Did you feel more confident in the lab having watched the instructional videos before attending lab? Strongly agree

Agree

Neutral

Disagree

Strongly Disagree

36%

45%

16%

3%

0%

6. In your opinion, should the use of instructional videos in the lab be expanded? Yes

No

86%

14%

The results of the CHEM 126 student survey on the use of the Laboratory Techniques videos (Table 3) showed surprisingly similar results to the questions asked in the VIU survey (Table 2). To the statement “Rate your level of agreement with the following statement: I have a deeper understanding of the lab material because of the instructional videos”, in both surveys 83% of the respondents either chose “Strongly agree” (12% vs. 13%) or “Agree” (71% vs. 70%). A strong majority of students in both surveys (~85%) also chose “Strongly agree” (36% vs 27%) or “Agree” (45% vs. 65%) to the statement “Did you feel more confident in the lab having watched the instructional videos before attending lab?”. The SFU study showed that a strong majority of the responding students found the videos to be helpful and thought that there should be an increased use of the Laboratory Techniques videos in the course. The instructional videos were developed to be an additional tool that can be used to engage the students with new material and this is reflected in the student survey. For one of the techniques covered in the Laboratory Techniques experiment in CHEM 126 (SFU), the information is provided to the students using four methods (20). Our survey examined which method of instruction the students preferred for the technique of manual titration. The students were exposed to the material by: (1) watching a demonstration by the instructor during the Laboratory Techniques experiment, (2) watching a video on manual titration before the experiment, (3) reading through a PowerPoint on manual titration during the experiment and (4) reading the Laboratory Techniques manual during the experiment. The survey results (Table 2) shows that the preferred method of instruction was watching the instructor perform a demonstration on the proper methods of the technique. This was followed by viewing the instructional videos, then the PowerPoint slides and finally referring to the technique manual. This reiterated that respondents still prefer to learn face-to-face with the instructor,

170 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

but that the instructional videos were preferable to PowerPoint slides and the laboratory manual. Comments provided by the students on the Laboratory Techniques videos include “It’s hard to visualize how things will look when set up without the videos, especially at the beginning of the course. They provided a sense of relief”, and “It allows students to observe described procedures in action, providing clarity that could otherwise be lost in wording”. Previous feedback comments included “The technique videos were very helpful because they showed exactly what we had to do or not do. I could watch it as many times as I needed and I was able to retain the information better than from just a document” and “The videos were an excellent resource. I referred to them before many labs and they were a great study tool for the practical midterm.” The effectiveness of the Instrumentation videos used in the analytical laboratory course (SFU: CHEM 215) was investigated through a voluntary online student feedback survey (Table 4) of students enrolled in the spring 2016 semester. Of the 64 students enrolled in the course, 11 completed the survey.

Table 4. Summary of student survey for CHEM 215 (SFU) 1. Did you watch any of the instructional videos? Yes

No

100%

0%

2. How helpful were the instructional videos? Very helpful

Somewhat helpful

Not at all helpful

36%

64%

0%

3. In this course the experimental procedure is presented to you in two ways, either provided in the lab manual or through instructional videos. Please indicate which method you find more effective in preparing you for the experiment. Reading the procedure in the lab manual

Watching the procedure in the instructional videos and taking notes

36%

64%

4. Which method of experimental procedure delivery do you prefer? Reading the procedure in the lab manual

Watching the procedure in the instructional videos and taking notes

55%

45% Continued on next page.

171 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Table 4. (Continued). Summary of student survey for CHEM 215 (SFU) 5. Did you feel more confident in the lab having watched the instructional videos before attending lab? Strongly agree

Agree

Neutral

Disagree

Strongly Disagree

18%

55%

18%

9%

0%

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

6. Rate your level of agreement with the following statement: I have a deeper understanding of the lab material because of the instructional videos. Strongly agree

Agree

Neutral

Disagree

Strongly Disagree

9%

45%

18%

18%

9%

7. In your opinion, should the use of instructional videos in the lab be expanded? Yes

No

82%

12%

Despite the small number of respondents, the feedback from the CHEM 215 student survey on the Instrumentation videos (Table 4) provided insight into the use and effectiveness of these videos. All respondents thought the videos were either “Very helpful” or “Somewhat helpful”. A majority of the respondents felt that using the videos to explain the proper use of the instrumentation was a more effective method than reading the steps in the laboratory manual, though if given the choice between the two methods the results were almost split. This was not unexpected as the replacement of the instruction in the laboratory manual by the videos meant the students no longer had access to step by step instructions, rather they had to develop them. Although developing the steps allows for a deeper understanding of how to operate the instrument, it requires less effort to simply read the instructions. The success of the CHEM 215 videos is seen in the majority of respondents (73% and 52% respectively) who felt more “confident in the lab having watched the instructional videos before attending lab” and had “a deeper understanding of the lab material because of the instructional videos”. Over 80% of the respondents support that “use of instructional videos in the lab be expanded”. Comments provided on the use of the Instrumentation videos include “I do better with visual learning so seeing the actions in progress is more effective for me as a student to learn about than reading instructions in the manual”. Previous collected feedback comments include “I have found the demonstration videos made for the analytical chemistry experiments to be very helpful. They have made the lab procedures more comprehensible and easier to do in the laboratory” and “The videos prevented a lot of confusion when I came to lab. I felt more confident about performing the experiment with the machine to be used”.

172 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

The effectiveness of the Laboratory Safety and Experimental Procedure videos used in the advanced inorganic laboratory course (SFU: CHEM 336) was investigated through another voluntary online student feedback survey (Table 5) of the students enrolled in the spring 2016 semester. Of the 26 students enrolled in the course, 12 completed the survey. The results of the survey of the students in Chem 336 (Table 5) of the use of the Laboratory Safety and Experimental Procedure videos highlighted the strength of the traditional education methods routinely used, but also the effectiveness of the instructional videos. A strong majority of the respondents (92% and 83%) felt that an instructor demonstration is more effective in preparing them for the experiments, and 83% and 67% of the respondents preferred the laboratory demonstration over the instructional video. When the role of the instructional video by itself was examined, its effectiveness became evident. A strong majority (75%) of respondents thought the instructional videos were helpful, with 83% strongly agreeing or agreeing that they where “more confident in the lab having watched the instructional videos before attending lab”. A similar result was obtained for the statement “Rate your level of agreement with the following statement: I have a deeper understanding of the lab material because of the instructional videos.”

Table 5. Summary of student survey for CHEM 336 (SFU) 1. Did you watch any of the instructional videos? Yes

100%

No

0%

How helpful were the instructional videos? Very helpful

Somewhat helpful

Not at all helpful

75%

25%

0%

2. The instructions on the safe use of the Schlenk line, fume hood and gas regulator were presented to you in two ways: (1) the instructor/TA demo and (2) the Instructional videos. Which method did you find more effective in preparing you for the experiment? Instructor demo

Instructional videos

92%

8%

3. Which method of experimental procedure delivery do you prefer? Instructor demo

Instructional videos

83%

17%

Continued on next page.

173 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Table 5. (Continued). Summary of student survey for CHEM 336 (SFU) 4. The instructions on experimental procedure for the Cr Acetate experiment were presented to you in two ways: (1) the instructor/TA demo and (2) the Instructional videos. Which method did your find more effective in preparing you for the experiment? Instructor demo

Instructional videos

83%

17%

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

5. Which method of experimental procedure delivery do you prefer? Instructor demo

Instructional videos

67%

33%

6. In general did you feel more confident in the lab having watched the instructional videos before attending lab? Strongly agree

Agree

Neutral

Disagree

Strongly Disagree

50%

33%

17%

0%

0%

7. Rate your level of agreement with the following statement: I have a deeper understanding of the lab material because of the instructional videos. Strongly agree

Agree

Neutral

Disagree

Strongly Disagree

25%

58%

17%

0%

0%

8. In your opinion, should the use of instructional videos in the lab be expanded? Yes

92%

No

8%

The success of the CHEM 336 videos is supported by the 92% of respondents who felt that “the use of instructional videos in the lab [should] be expanded”. Comments provided by the students included “they were great”, “I think it’s great, they are short and concise“ and they liked the fact that “the videos can be replayed, so steps can be reviewed [for] better visualization of the lab and apparatus going to be supplied”.

Challenges and Successes of Developing Instructional Videos Through the successes and setbacks experienced in the development of the instructional videos at SFU and VIU, and the feedback from students, some “best practice” methods emerged. Students reported that videos that are short, clear and concise are the most effective. Students prefer videos that are 2-3 minutes long and focused on a single topic that is clearly explained. 174 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

The quality of the audio and video was also important to the students. Videos that have high resolution recording, high quality audio, and a simple cinematic style with quality video production resonate most with students. The sound of the voice and speaking style of the narrator needs to be considered when developing the videos. The audio on the initial videos developed at SFU have all been rerecorded, as the original voice was found to be too monotone and the source of most of the negative feedback from the students. The videos are now narrated in a more engaging speaking style. This change has received positive feedback from students. The videos should also include closed captioning in order to increase accessibility of the videos to the hearing impaired and English as a second language students. Students also prefer to see videos that are true to what they will experience in the laboratory. There is a sense of familiarity if the videos are filmed in the same laboratory space with the same equipment that the students will use. Videos developed by an institution also allow for the content to be tailored to the specific expectations of the institution (5). A large amount of effort is required to develop these videos; therefore steps should be taken to promote their sustainable use. All colleagues who instruct the course(s) which use the videos should be consulted to ensure their satisfaction in the product and their support for using them. Care should also be taken to avoid any information that will date the video, thus decreasing its “shelf-life”. Popular catchphrases, colour schemes, and sound effects that maybe associated with a specific time should be avoided. This may “age” the video and decrease its appeal in the future. Content specific information, such as course and semester codes, dates, times and page numbers should be avoided as these details can change with updates to the course (17).

Conclusion The laboratory videos developed in the chemistry departments of both SFU and VIU have been shown to be an effective tool in improving our students’ laboratory learning experience. Videos were developed which cover five main categories and were used in first, second and third year courses. Most videos were developed to enhance existing teaching methods, while one series of videos was used to completely replace the standard method. On average 87% of survey respondents at SFU thought the use of laboratory videos should be expanded. The majority of the respondents from both universities (76%) felt that they had a deeper understanding of the experiment/theory due to the videos, with a similar amount (79%) indicating that they felt more confident about the experiment having watched the instructional videos. Our study also highlights that many students prefer the traditional methods of learning the material face-to-face in the laboratory setting, but that the instructional videos are a valuable resource that they can use in their discovery and review of the material. Instructional videos should not replace conventional teaching methods, but do provide another effective tool that can be employed as we strive to better educate our students. 175 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

Acknowledgments SFU: This work was supported by four Teaching and Learning Development Grants from the Institute for the Study of Teaching and Learning in the Disciplines (ISTLD) and the Teaching and Learning Centre (TLC) at Simon Fraser University (25). We wish to thank Gilberto Martinez and Teresita Barbou for their work on the filming and editing of the videos. VIU: The production of laboratory video tutorials was supported by a Vancouver Island University Centre for Innovation and Excellence Learning Innovation and Enhancement in Teaching (LITE) Grant. Student survey and analytic data research methods were performed according to Vancouver Island University Research Ethics Board guidelines (17). All authors wish to thank the students who have tested the videos and have provided us with valuable feedback.

References 1. 2. 3. 4.

5.

6.

7.

8.

9.

Reid, N.; Shah, I. The role of laboratory work in university chemistry. Chem. Educ. Res. Pract. 2007, 8, 172–185. DOI: 10.1039/b5rp90026c. DeMeo, S. Teaching chemical technique. J. Chem. Educ. 2001, 78, 373–379. DOI: 10.1021/ed078p373. Bopegedera, A. M. R. P. Putting the laboratory at the center of teaching chemistry. J. Chem. Educ. 2011, 88, 443–449. DOI: 10.1021/ed100045z. Towns, M.; Harwood, C. J.; Robertshaw, M. B.; Fish, J.; O’Shea, K. The digital pipetting badge: A method to improve student hands-on laboratory skills. J. Chem. Educ. 2015, 92, 2038–2044. DOI: 10.1021/acs.jchemed. 5b00464. Jordan, J. T.; Box, M. C.; Eguren, K. E.; Parker, T. A.; Saraldi-Gallardo, V. M.; Wolfe, M. I.; Gallardo-Williams, M. T. Effectiveness of studentgenerated video as a teaching tool for an instrumental technique in the organic chemistry laboratory. J. Chem. Educ. 2015, 92, 141–145. DOI: 10.1021/acs.jchemed.5b00354. Tasker, R. ConfChem conference on interactive visualizations for chemistry teaching and learning: Research into practice-visualizing the molecular world for a deep understanding of chemistry. J. Chem. Educ. 2016, 93, 1152–1153. DOI: 10.1021/acs.jchemed.5b00824. Velazquez-Marcano, A.; Williamson, V. M.; Ashkenazi, G.; Tasker, R.; Williamson, K. C. The use of video demonstrations and particulate animation in general chemistry. J. Sci. Technol. 2004, 13, 315–323. DOI: 10.1023/B:JOST. 0000045458.76285.fe. Conole, G.; Dyke, M. What are the affordances of information and communication technologies? ALT: Res. Learn. Tech. 2004, 12, 113–124. DOI: 10.1080/0968776042000216183. McGoldrick, B.; Watts, J. S.; Economou, K. U.S. postsecondary faculty in 2015: Diversity in people, goals and methods, but focused on students. http://postsecondary.gatesfoundation.org/wp-content/uploads/2015/02/USPostsecondary-Faculty-in-2015.pdf (accessed March 22, 2016) 176

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch009

10. Conole, G.; Dyke, M.; Oliver, M.; Seale, J. Mapping pedagogy and tools for effective learning design. Comput. Educ. 2004, 43, 17–33. DOI: 10.1016/ j.compedu.2003.12.018. 11. Tabata, L. N.; Johnsrud, L. K. The impact of faculty attitudes toward technology, distance education, and innovation. Res. High. Educ. 2008, 49, 625–646. DOI: 10.1007/s11162-008-9094-7. 12. Day, J. A.; Foley, J. D. Evaluating a web lecture intervention in a human–computer interaction course. IEEE T. Educ. 2006, 49, 420–431. DOI: 10.1109/TE.2006.879792. 13. Active Presenter Home Page. http://atomisystems.com/activepresenter/ (accessed July 4, 2016) 14. Audacity Home Page. http://www.audacityteam.org/ (accessed July 4, 2016) 15. Windows Live Movie Maker Support Page. https://support.microsoft.com/ en-us/help/14019/windows-movie-maker-edit-movies (accessed July 4, 2016) 16. Adobe Premiere Elements Home Page. http://www.adobe.com/ca/products/ premiere-elements.html (accessed July 4, 2016) 17. Key, J.; Paskevicius, M. Investigation of video tutorial effectiveness and student use for general chemistry laboratories. J. App. Learn. Technol. 2015, 5, 14–21. 18. SFU LONCAPA Home page. http://www.sfu.ca/loncapa/ (accessed March 25, 2016) 19. SFU Canvas Home page. http://www.sfu.ca/canvas.html (accessed March 25, 2016) 20. Canal, J. P.; Lowe, J.; Fong, R. Improving Students’ Practical Laboratory Techniques through Focused Instruction and Assessment. In Technology and Assessment Strategies for Improving Student Learning in Chemistry; Schultz, M., Schmid, S., Holme, T., Eds.; Symposium Series 1235; American Chemical Society: Washington, DC, 2016; Chapter 8. 21. Meloan, C. E. The use of tape recorders, cartridged films, and real samples in laboratories. J. Chem. Educ. 1971, 78, 139–141. 22. Kempa, R. F.; Palmer, C. R. The effectiveness of video-tape recorded demonstrations in the learning of manipulative skills in practical chemistry. Brit. J. Educ. Technol. 1974, 5, 62–71. DOI: 10.1111/j.14678535.1974.tb00623.x. 23. McNaught, C.; Grant, H.; Fritze, P.; Barton, J.; McTigue, P.; Prosser, R. The effectiveness of computer-assisted learning in the teaching of qualitative volumetric analysis in a first-year university course. J. Chem. Educ. 1995, 72, 1003–1007. DOI: 10.1021/ed072p1003. 24. Teo, W. T.; Tan, K. C. D.; Yan, Y. K.; Teo, Y. C.; Yeo, L. W. How flip teaching supports undergraduate chemistry laboratory learning. Chem. Educ. Res. Pract. 2014, 15, 550–567. DOI: 10.1039/C4RP00003J. 25. SFU Teaching and Learning Development Grant Home Page. http:// www.sfu.ca/tlgrants.html (accessed March 25, 2016)

177 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

Chapter 10

Using the ACS Anchoring Concepts Content Map (ACCM) To Aid in the Evaluation and Development of ACS General Chemistry Exam Items Jessica J. Reed,1 Cynthia J. Luxford,2 Thomas A. Holme,3 Jeffrey R. Raker,4 and Kristen L. Murphy*,1 1Department

of Chemistry & Biochemistry, University of Wisconsin-Milwaukee, Milwaukee, Wisconsin 53211, United States 2Department of Chemistry & Biochemistry, Texas State UniversitySan Marcos, San Marcos, Texas 78666, United States 3Department of Chemistry, Iowa State University, Ames, Iowa 50011, United States 4Department of Chemistry, University of South Florida, Tampa, Florida 33620, United States *E-mail: [email protected]

The ACS Anchoring Concepts Content Map (ACCM) for general chemistry provides an increasingly detailed framework for examining specific components of chemistry content across ten big ideas of chemistry. The ACCM can serve as a valuable tool for evaluating the content coverage of an existing assessment, and can aid in the creation of new assessment materials by allowing assessment creators to ensure items covering a broad range of chemistry topics are included. Previous work aligned ACS general chemistry exam items to the ACCM and identified locations on the map lacking exam item coverage. This chapter presents insights of how committees of practitioners used the ACCM as a framework to create new ACS general chemistry exams that aimed to eliminate some of the gaps in assessment coverage found in previous ACS exams. Gaps related to topics within the big ideas of bonding and energy and thermodynamics will be highlighted. The results indicate a modest shift that begins to eliminate coverage gaps © 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

in ACS general chemistry exams when the ACCM is used as a guide for writing exam items. These results also provide implications for chemistry educators about the usefulness of the ACCM when creating assessment materials.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

Introduction Introductory level chemistry courses are often gateway courses for many other science, technology, engineering, and math (STEM) courses (1, 2). In many cases, a student’s only exposure to chemistry concepts comes from what is taught and assessed in these foundational general chemistry courses. In this regard, it may be worth considering whether what is being assessed is in line with the expectations of what students ought to know after taking these courses. While it would be counterproductive to assess every nuance of chemistry content taught, it could be valuable to understand how the content taught and content assessed intersect. In this regard, the questions of “What do we assess?” versus “What do we want to assess?” become important to address. In order to bridge the gap between these two questions, it is necessary to have some means to determine what we are currently assessing in our chemistry courses and make a comparison as to whether that is what we actually want to assess. In the K-12 sector this is often accomplished through the use of standards, for example the Next Generation Science Standards (NGSS) (3). While the use of standards at the collegiate level is not anticipated, there are a growing number of calls for assessment accountability in higher education (4–6). In this regard, the use of a framework to bridge course content with assessment endeavors may be beneficial to many voluntary mechanisms institutions and departments are using to address issues of teaching and learning accountability (7). In chemistry, the development of such a framework, the Anchoring Concepts Content Map (ACCM) (8), provides a mechanism for chemistry faculty and departments to align their assessment materials to an externally generated content framework for measurement of learning outcomes. Here we explore the use of the ACCM framework to aid in the development of chemistry exam content when a mismatch between what we assess and what we want to assess occurs.

Development and Structure of the Anchoring Concepts Content Map (ACCM) The ACCM project was developed by the American Chemical Society Examinations Institute (ACS-EI) as a means to provide instructors enhanced assessment abilities across the undergraduate chemistry curriculum (8, 9). In this regard, the goal was to create a series of interlinking content maps that span the undergraduate chemistry curriculum to which ACS Exam items and instructors’ personally generated assessment items could be aligned for the purpose of measuring learning outcomes. It is important to note that the ACCM is not meant to serve as a set of standards for any chemistry curriculum. Rather, the ACCM should be construed as a tool available to faculty and administrators for the 180 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

purpose of understanding and evaluating the content taught and assessed within their courses and departments. Content maps for all subdisciplines of the undergraduate curriculum (general, organic, analytical, inorganic, physical, and biochemistry) are currently in various stages of development and publication (10–12), however, the work in this chapter will focus on the ACCM for general chemistry (10, 11). The ACCM for general chemistry is readily available for use by chemistry practitioners and researchers (10, 11). The development of the various subdiscipline content maps has been conducted through numerous workshops with chemistry faculty and instructors. Murphy, et. al. (8, 9), provides a timeline and a more detailed description of how the structure of the ACCM was developed. The ACCM was developed using backward design (13) and is constructed in a hierarchical structure with ten broad, subdiscipline independent Big Ideas or Anchoring Concepts comprising the top-tier (Level 1) and spanning to fine grained subdiscipline specific Content Details (Level 4).

Figure 1. Hierarchical Structure of the ACCM. Figure 1 displays the hierarchical structure of the map and Table 1 displays the titles of the ten Big Ideas and the Anchoring Concepts that support them. The Big Ideas and Enduring Understanding statements, Levels 1 and 2, respectively, are the same across all subdisciplines, while the subdiscipline articulations and content details, Levels 3 and 4, respectively, are specific to the subdiscipline. Therefore, there are a fixed number of Big Idea and Enduring Understanding statements, but the number of subdiscipline articulations and content details vary depending upon the subdiscipline. An example of how the ACCM varies at the levels of 181 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

subdiscipline articulation and content details for general and organic chemistry is shown in Table 2. An important feature of the ACCM is the fact that the Level 1 and Level 2 statements remain the same across all subdisciplines because it provides a means for departments to set learning objectives that span the entire undergraduate chemistry curriculum and then have an external reference to align course content and assessments. Additionally, the ACS-EI has aligned general chemistry ACS Exam items (14) to the ACCM so ACS Exams users may examine student performance on specific content details or make comparisons of content coverage between courses. As more subdiscipline content maps are completed and ACS Exam items are aligned to the maps, the utility of the ACCM for longitudinal comparison of curriculum and assessment materials will increase.

Table 1. Big Ideas and Anchoring Concepts of ACCM (level 1: subdiscipline independent) Big Idea

Anchoring Concept

I. Atoms

Matter consists of atoms that have internal structures that dictate their chemical and physical behavior.

II. Bonding

Atoms interact via electrostatic forces to form chemical bonds.

III. Structure and Function

Chemical compounds have geometric structures that influence their chemical and physical behaviors.

IV. Intermolecular Interactions

Intermolecular forces—electrostatic forces between molecules—dictate the physical behavior of matter.

V. Chemical Reactions

Matter changes, forming products that have new chemical and physical properties.

VI. Energy and Thermodynamics

Energy is the key currency of chemical reactions in molecular scale systems as well as macroscopic systems.

VII. Kinetics

Chemical changes have a time scale over which they occur.

VIII. Equilibrium

All chemical changes are, in principle, reversible; chemical processes often reach a state of dynamic equilibrium.

IX. Experiments, Measurement, and Data

Chemistry is generally advanced via experimental observations.

X. Visualization

Chemistry constructs meaning interchangeably at the particulate and macroscopic levels.

182 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Table 2. Example of all four levels within General and Organic Chemistry (showing the difference beginning at level 3)

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

Level

General Chemistry

Organic Chemistry

1

Matter consists of atoms that have internal structures that dictate their chemical and physical behavior.

2

Electrons play the key role for atoms to bond with other atoms

3

For a neutral atom there are as many electrons as there are protons, but the electrons can be categorized as core (inner) and valence (outer) electrons

Electrons play a role in understanding the relative stability of resonance structures.

4

Valence electrons, which determine the properties of elements, are correlated with the groups in the periodic table

Stabilization of anions helps to explain pKa values and relative acidities of protons.

Development of ACS Exams ACS Exams are developed in a grassroots fashion, meaning that the ACS-EI does not specify what content must be included in an examination other than it must be appropriate for the level of chemistry to be tested (15). Committees of chemistry faculty and practitioners work together to generate exam items which are then trial tested and validated by student performance metrics. It takes approximately two years for an ACS Exam to be developed and released to the community. A timeline of this process is shown in Figure 2.

Figure 2. Exam development timeline. Because exam committees are comprised of practitioners, ACS Exams reflect the content deemed important to assess by the chemistry community as a whole. While exam committees are not given item specifications to follow when constructing the exam, there is often a tendency for exam content coverage to mimic recently released ACS Exams from the same domain. This can lead to gaps in assessment of certain concepts as shown by Luxford and Holme (16). In order to address these gaps in assessment and avoid perpetuation of conceptual holes, exam committees ought to be made aware of the existence of such gaps and provided the opportunity to remedy these gaps by writing items that assess frequently overlooked content details. Use of the ACCM as a tool to guide these efforts may prove beneficial. 183 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Methods

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

Exam Item Alignment ACS Exams provide a unique snapshot of what the chemistry community values in assessment because the exams are created by committees and measure the content the committee deems important. However, released exams may not show the full picture of the content the exam writing committee wanted to assess because items that do not perform well are removed following trial testing. These items that never make it to a released exam are useful artifacts to analyze to understand the relation between what was actually tested versus what was intended to be tested. In the current research project, it is postulated that exam writing committees that were made aware of the conceptual holes found in prior ACS Exams (16), and used the ACCM to guide the creation of new exam content, were able to generate assessment items that tested content that had previously been overlooked. Previous work aligned released ACS general chemistry exams to the ACCM (14) and identified areas, particularly in Big Idea II (Bonding) and VI (Energy and Thermodynamics), where few ACS Exam items were being aligned (16). The current project examined how content coverage from released and unreleased exam items from the ACS First-Term General Chemistry Exams from 2012 (GC12F) and 2015 (GC15F) compare because the GC15F exam was developed with aid of the ACCM and the GC12F was not. The project also examined content coverage from the trial tests developed for the 2017 full-year General Chemistry Exam (GC17) because the exam committee used the ACCM during the exam development process. This analysis was conducted by a research team from the ACS EI in order to understand more thoroughly how use of the ACCM during the exam development process may influence exam content coverage and aid in the elimination of conceptual holes in ACS Exams. Assessment alignment can occur in a variety of ways and can be useful for identifying how assessment content and curriculum interect (17–19). In order to evaluate the content coverage of these exams for research purposes, the exam items were first aligned to the ACCM by an experienced rater. The alignment process involved looking at individual items and determining, first, which Big Idea matched the item, then reading through the Enduring Understanding, Subdisciplinary Articulation, and Content Details statements of that Big Idea to determine where the item best aligned. So, for example, an item asking for the number of neutrons in an atom of fluorine would first be aligned to Big Idea I (Atoms) and then aligned to Enduring Understanding “A” about the number of protons in the nucleus giving rise to an atom’s unique identity. From here, Subdisciplinary Articulation “2” is selected and then Content Detail “a” regarding protons and neutrons summing to contribute the mass of an atom is selected as the final coordinate for where the item belongs. Thus, the item now has four coordinates indicating its location on the ACCM. In this example, the item has been aligned to the location “I A 2 a” on the map. A visual representation of the alignment process described in this example is illustrated in Figure 3. Every effort was made to align items to statements at the Content Details (most specific) level, which provided each item with a set of four “coordinates” indicating its location on the ACCM, but on occasion some items were unable to be aligned 184 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

to the content specificity of Level 4 statements. Also, sometimes an item fit in multiple locations on the ACCM, in which case it was aligned to multiple coordinates. When this occurred, there was no priority assigned to multiple alignment locations for one item, treating the multiple locations of equivalent value. While the alignment process may appear tedious at first, with practice it can be completed with ease.

Figure 3. Example of the alignment process for a general chemistry assessment item.

The item alignments for the current project were completed by an experienced rater and then a subset of the items was aligned by a team of three additional raters in order to ensure consistency and accurate alignment. In situations where the rating team did not agree on the alignment location for an item, the team discussed possible alignment locations until 100 percent agreement was reached. In total, 424 released and unreleased items were aligned in this analysis and a distribution of items by exam is shown in Table 3. 185 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Table 3. Distribution of Analyzed Items by Exam Type

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

ACS Exam Items Aligned to ACCM Exam

Released Items Analyzed

Unreleased Trial Items Analyzed

GC12F

70

72

GC15F

70

72

GC17

n/a

140

Total

140

284

Using the ACCM To Create ACS Exam Content As of this writing, two general chemistry exam committees have used the ACCM to aid in the development of ACS Exams. The committee chairs for the GC17 and the GC15F exams shared their experiences using the ACCM to assist in exam development. These experiences were shared in the form of written responses to questions posed by the authors and each committee chair provided written consent to use her name and quotes in this manuscript. When asked about how their respective committees used the ACCM, GC15F exam committee chair Sharmistha Basu-Dutt described a process of first becoming familiar with the ACCM and then “We looked at a couple of old [ACS] exams and aligned them to the ACCM […]. After we agreed on how each question fit the ACCM topics, we generated a list to see if there were topics that were over-represented or under-represented in previous [ACS] exams.” She went on to explain that the committee members were given the opportunity to develop their own questions, but were “charged to fit their question to specific topics and subtopics on the ACCM.” The committee members’ questions were then compiled and all of the items relating to a particular Big Idea and Enduring Understanding were considered at the same time making the process “very streamlined.” In regard to using the ACCM as opposed to following the content distributions previous exam committees had used, GC17 exam committee chair Yasmin Patell stated: “The original plan was to explore the general chemistry topic grid that had been used by previous exam committees as a blue-print […]. However, once the committee members became familiar with the ACCM, it became clear that the ACCM provided a far superior template that offered genuinely comprehensive coverage of the general chemistry curriculum.” Basu-Dutt also commented on this by saying “[…] the ACCM helped us have a good understanding of the general chemistry curriculum and we were able to use our time efficiently to develop a test that had a good balance of topics included.” Patell also pointed out that using the ACCM “brought about some helpful self-realization for many committee members such as, what topics do I cover in my own day-to-day teaching, and should I consider expanding/changing my topic 186 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

selections, or should I reassess how much/little time I spend on different areas?” In this regard, the ACCM can be a valuable tool for instructors to use in their own classrooms. Use of the ACCM to create assessment materials is certainly not limited to ACS Exams, and in fact, instructors may find it beneficial to align their own course exam items to the ACCM to identify what they test and how that compares to what they want to test. Additionally, the Content Details (Level 4) statements of the ACCM may help instructors generate content for individual exam items. It is not expected that the ACCM would serve as a mandated set of test blueprints, but rather as an external framework for instructors to use to create, analyze, and compare assessment materials within and across courses. Overall, both committee chairs felt that the ACCM helped them to see where previous assessments may have fallen short in terms of breadth of content coverage and allowed them to remedy these shortcomings by creating assessment items to fill the gaps. It appears that lack of awareness of the ACCM and its potential uses for test development are key barriers to its implementation. Both committee chairs described that their exam committees agreed unanimously to use the ACCM once they became familiar with its design and usefulness. It is important to note that the ACS-EI does not prescribe the use of the ACCM for the creation of new exams, rather it is a tool available to assist exam development committees as they write assessment items. Content coverage decisions are always made by the exam development committees based on the collective experience of the committee.

Results and Analysis The GC15F and GC17 exam committees used the ACCM to guide development of their respective exams. Currently, the GC15F has been released for purchase and use by the chemistry community, but the GC17 exam is still in stages of trial testing and development. Therefore, the results presented for the GC17 exam will represent all unreleased trial items and are considered independently from the GC15F results because the exams have different content coverage. A closer look at alignments for the GC15F and GC17 exams reveals that use of the ACCM for test item creation showed a modest shift toward elimination of gaps in assessment coverage. Because ACS Exams are secure, copyrighted exams, the specific items created to address these conceptual holes cannot be shown. It is important to note that the y-axis represents the number of alignments rather than the number of items because some items were aligned to more than one location in the ACCM. The figures presented here will show all written items for GC12F, GC15F and GC17 exams. For the first-term exams (GC12F and GC15F), the total number of exam items written (prepared for trial testing) is the sum of the released and unreleased items. The released items are on the exam, and the unreleased items were trial items that were not selected for the released test. Therefore, the number of released items in one content category can exceed the number of unreleased items as more items were selected for the released test compared to those that were not. In general, approximately 50 percent of the items written by an exam committee make it onto the released version of the test. 187 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

Assessing Big Idea II: Bonding Luxford and Holme (16) identified that the Big Idea of Bonding was one of the least tested across all ACS general chemistry exams analyzed, and pointed to conceptual holes in assessment related to Enduring Understanding statements 2D and 2G about bond breaking requiring energy input and metallic bonding, respectively. Comparison of released and unreleased items associated with GC12F and GC15F exams across the Enduring Understanding statements of Big Idea II is shown in Figure 4. The GC12F exam was not developed with the aid of the ACCM while the GC15F exam was. This comparison revealed that there were still conceptual holes in the assessment, particularly for statement 2D, but growth in the number of items created addressing Enduring Understanding statements 2A (7 items total for GC15F compared to 2 for GC12F) and 2F (3 items total for GC15F compared to 1 for GC12F). The lack of items on the GC15F exam assessing Enduring Understanding 2D could be due to the fact that a similar statement is found in Big Idea VI related to energy and thermodynamics.

Figure 4. Distribution of released and unreleased items from GC12F and GC15F across Enduring Understanding statements for Big Idea II: Bonding. The distribution of unreleased trial items for the GC17 exam across the Enduring Understanding statements of Big Idea II is shown in Figure 5. Because it is yet to be determined how many of these items will make it to the released exam, a comparison between the distribution of GC17 unreleased trial items and the distribution of items for a released full-year general chemistry exam would not be very meaningful. What can be observed, however, from the GC17 item distribution across the Big Idea of Bonding is that while the greatest number of items are still found within Enduring Understanding 2C, as also observed in the first-term exam distributions, relatively untested topics are beginning to appear. For example, statement 2D about bond breaking requiring an energy input has one trial item and 2E regarding molecular orbital theory has three trial items. How these items will perform during trial testing and whether they will make it to the final released version of the exam is still unknown, but the fact that items 188 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

are at least being constructed about these topics suggests that there has been acknowledgment of at least some of the conceptual holes present in previous assessments.

Figure 5. Distribution of unreleased trial items from GC17 across Enduring Understanding statements for Big Idea II: Bonding. Assessing Big Idea VI: Energy and Thermodynamics An additional area where conceptual holes were noted was in Big Idea VI: Energy and Thermodynamics (16). There were several Level 2 statements with few assessment items aligned to them, however, it could be argued that lack of items assessing net change in energy of a system (6A) and energy input for bond breakage (6D) are the most disconcerting considering their importance and relevance to the general chemistry curriculum. Figures 6 and 7 show various distributions of items across the Enduring Understanding statements of Big Idea VI. The idea that bond breaking requires an energy input was not readily addressed in Big Idea II, however, items aligning with this content are found in Enduring Understanding 6D. In Figure 6, it is observed that only one item was written about this topic for the GC12F exam, and that item did not make it to the final released version of the exam, however, for the GC15F exam a total of four items were written to address this topic and two made it to the released exam. Figure 7 shows that two trial items were written for the GC17 exam regarding the energy input required when bonds break (6D). While there are still some Enduring Understanding statements without assessment items, the distribution of GC17 unreleased trial items shows fairly comprehensive coverage of Big Idea VI. Additionally, the GC17 trial items contain two items related to the net change in energy of a system (6A) which has historically been tested very infrequently (16). Concepts related to energy and thermodynamics at the macroscopic scale such as harnessing energy via devices (6F) and concepts related to implications of nuclear chemistry (6I) have historically not been assessed by ACS Exams (16), likely because of limitations due to the number of items on these exams. While 189 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

it is disappointing that items related to these applications of chemistry did not appear in trial items, it is not unexpected.

Figure 6. Distribution of released and unreleased items from GC12F and GC15F across Enduring Understanding statements for Big Idea VI: Energy and Thermodynamics.

Figure 7. Distribution of unreleased trial items from GC17 across Enduring Understanding statements for Big Idea VI: Energy and Thermodynamics. Distribution of Exam Items across the ACCM Because there is a finite number of items on ACS Exams, there may be concern about incorporating assessment items to fill in content coverage gaps at the detriment to assessment of another topic. Comparison of the distributions of item alignments across all ten Big Ideas of the ACCM for GC12F and GC15F 190 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

(Figure 8) reveals minor differences congruent with shifting some items from one content area to another in order to address conceptual holes, but not so much as to warrant concern. Additionally, Figure 9 shows all ten Big Ideas as represented in the GC17 unreleased trial items. Still, it is important to consider, at least in the realm of ACS Exams, the notion of whether what we want to assess is worth the sacrifice of what we are currently assessing. It is speculated that in some instances, for example metallic bonding (2G), while an exam committee may want to include a topic on an exam, the content coverage that would need to be sacrificed does not make it worth doing.

Figure 8. Distribution of released GC12F and GC15F exam items across the ten Big Ideas of the ACCM. The GCF exam series is designed for a first-semester general chemistry course and therefore intentionally does not include the topics of kinetics or equilibrium.

It is important to note that just because a topic is not assessed does not mean that it is not taught or valued in the chemistry curriculum. Additionally, some instructors may choose to test these topics more frequently in their own course exams compared to how frequently they appear on ACS Exams. As described earlier, the limited number of items included on an ACS Exam means that not all topics can be given equal representation on exams. Overall, modest shifts in the number of items developed to test topics previously overlooked in ACS Exams was observed when exam development committees used the ACCM to guide exam construction. Dramatic changes to exam content coverage were not expected due to the limited number of items on ACS Exams and the nature of content to be assessed. These modest shifts in exam content coverage suggest a growing awareness of potential gaps in assessment coverage and a need to address them.

191 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

Figure 9. Distribution of GC17 unreleased trial items across the ten Big Ideas of the ACCM.

Summary and Implications In general, the use of the ACCM to aid in exam construction proved to be a valuable experience for two ACS general chemistry exam committees. The researchers’analysis of items created by these committees suggests that as the committees became aware of important gaps in content coverage on previous ACS general chemistry exams they sought to create new exams that provide a more comprehensive assessment of the curriculum. The results of these endeavors showed modest increases in the number of items being constructed about important topics related to bonding and energy and thermodynamics that had rarely been assessed on previous ACS general chemistry exams. The implications of this work are twofold. First, there are implications for ACS Exams developers and second there are implications for instructors generating their own assessment materials. It is important to revisit the question of “What are we testing, and how does that align with what we want to test?” From the perspective of ACS Exams development, the use of the ACCM by exam writing committees creates greater awareness of what is being tested and how it fits within the curriculum. This creates the potential for more comprehensive content coverage on exams. It also allows committees to better judge the cost of including some topics on an exam while excluding others. From the perspective of instructors, the use of the ACCM when creating assessment materials has a variety of implications. First, instructors are able to align their own previously created exam items to the ACCM to better understand their own patterns of assessment. Perhaps they may notice content areas that receive much of the focus of their assessments while other topics they value for their students to know are assessed very little; or perhaps they will notice that their assessments are well distributed across the topics that they value for student mastery. This ability for self-evaluation may provide opportunity to 192 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

enhance alignment between course content and assessment content. Secondly, the ACCM allows instructors to have an external framework to support creation of assessment items to measure content learning objectives. When writing an exam, the instructor can look to the Content Details (Level 4) statements on the ACCM for aid in determining the specific content to include in test items. Additionally, large courses with multiple sections and instructors may find it beneficial to use the ACCM when creating exams to ensure consistent content assessment between multiple exam forms. Finally, as more subdiscipline ACCMs are completed, a department may choose to use align assessment materials to the various subdiscipline ACCMs to evaluate programmatic assessment endeavors across the undergraduate chemistry curriculum. In summary, there is a lot of flexibility in how individual instructors and chemistry departments may choose to use the ACCM, but it is anticipated that by using the ACCM their teaching and assessment efforts will be aided. Overall, this statement by Yasmin Patell, GC17 exam committee chair, sums up the relevance of the ACCM to the chemistry community: “Simply put, the ACCM provides an invaluable pedagogical tool for chemistry teaching, learning, and assessment.”

References 1.

2.

3. 4. 5.

6.

7. 8.

Gasiewski, J. A.; Eagan, M. K.; Garcia, G. A.; Hurtado, S.; Chang, M. J. From gatekeeping to engagement: A multicontextual, mixed method study of student academic engagement in introductory STEM courses. Res. High. Educ. 2012, 53, 229–261, DOI: 10.1007/s11162-011-9247-y. Wright, A.; Provost, J.; Roecklein-Canfield, J. A.; Bell, E. Essessential concepts and underlying theories from physics, chemistry, and mathematics for “Biochemistry and Molecular Biology” majors. Biochem. Mol. Biol. Educ. 2013, 41, 302–308, DOI: 10.1002/bmb.20728. NGSS Lead States. Next Generation Science Standards:For States, By States. The National Academies Press: Washington, DC, 2013. Liu, O. L. Value-added assessment in higher education: A comparison of two methods. High. Educ. 2011, 61, 445–461. Liu, O. L. Outcomes assessment in higher education: challenges and future research in the context of voluntary system accountability. Educ. Meas. Issues Pract. 2011, 30, 2–9, DOI: 10.1111/j.1745-3992.2011.00206.x. Spellings, M. A Test of Leadership: Charting the Future of U.S. Higher Education; U.S. Department of Education, Education Publication Center: Jessup, MD, 2006. https://www2.ed.gov/about/bdscomm/list/hiedfuture/ reports/pre-pub-report.pdf (accessed August, 2016). Bretz, S. L. Navigating the landscape of assessment. J. Chem. Educ. 2012, 89, 689–691, DOI: 10.1021/ed3001045. Murphy, K.; Holme, T.; Zenisky, A.; Caruthers, H.; Knaus, K. Building the ACS Exams Anchoring Concept Content Map for undergraduate chemistry. J. Chem. Educ. 2012, 89, 715–720, DOI: 10.1021/ed300049w. 193

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

9.

10.

11.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

12.

13. 14.

15. 16.

17.

18.

19.

Zenisky, A. L.; Murphy, K. L Developing a Content Map and Alignment Process for the Undergraduate Curriculum in Chemistry. In Trajectories of Chemistry Education Innovation and Reform, 2013; Chapter 6, pp 79−91. Holme, T.; Murphy, K. The ACS Exams Institute undergraduate chemistry Anchoring Concepts Content Map I: General Chemistry. J. Chem. Educ. 2012, 89, 721–723, DOI: 10.1021/ed300050q. Holme, T.; Luxford, C.; Murphy, K. Updating the General Chemistry Anchoring Concepts Content Map. J. Chem. Educ. 2015, 92, 1115–1116, DOI: 10.1021/ed500712k. Raker, J.; Holme, T.; Murphy, K. The ACS Exams Institute undergraduate chemistry Anchoring Concepts Content Map II: Organic Chemistry. J. Chem. Educ. 2013, 90, 1443–1445, DOI: 10.1021/ed400175w. Wiggins, G. P.; McTighe, J. Understanding by Design. ACSD: Alexandria, VA, 2005. Luxford, C. J.; Linenberger, K. J.; Raker, J. R.; Baluyut, J. Y.; Reed, J. J.; De Silva, C.; Holme, T. A. Building a database for the historical analysis of the general chemistry curriculum using ACS General Chemistry exams as artifacts. J. Chem. Educ. 2015, 92, 230–236, DOI: 10.1021/ed500732q. Holme, T. A. Assessment and quality control in chemistry education. J. Chem. Educ. 2003, 80, 594–596, DOI: 10.1021/ed080p594. Luxford, C. J.; Holme, T. A. What do conceptual holes in assessment say about the topics we teach in general chemistry? J. Chem. Educ. 2015, 92, 993–1002, DOI: 10.1021/ed500889j. Webb, N. In Handbook of Test Development; Downing, S. M., Haladyna, T. M., Eds.; Lawrence Erlbaum Associates, Inc.: Mahwah, NJ, 2006; pp 155−180. Pinto, G. The Bologna Process and its impact on university-level chemical education in Europe. J. Chem. Educ. 2010, 87, 1176–1182, DOI: 10.1021/ ed1004257. Plaza, C. M.; Draugalis, J. R.; Slack, M. K.; Skrepnek, G. H.; Sauer, K. A. Curriculum mapping in program assessment and evaluation. Am. J. Pharm. Educ. 2007, 71, Article 20.

194 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Chapter 11

How Do Chemistry Educators View Items That Test Conceptual Understanding? Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011

Cynthia Luxford1 and Thomas Holme*,2 1Department

of Chemistry and Biochemistry, Texas State University, San Marcos, Texas 78666, United States 2Department of Chemistry, Iowa State University, Ames, Iowa 50011, United States *E-mail: [email protected]

The ability to test student understanding of chemistry faces a number of challenges that result from the multi-faceted nature of the science. Students are expected to be capable of dealing with quantitative aspects and much of the introductory content of college chemistry course is geared towards this component. Over the past 3 decades, interest in the achievement of conceptual understanding by students has increased. Attempts have been made to allow demonstration of such knowledge through traditional written exams. However, the nature of what constitutes effective test questions for this task has received less attention. A survey on testing for conceptual understanding was given to roughly 13,000 general chemistry instructors across the nation. Responses from 1800 faculty were recorded and the responses of 1519 faculty members were analyzed after cleaning the dataset. Faculty were asked to determine whether a series of 6 questions similar to ACS exam questions were measuring conceptual understanding. Immediately after their rankings, they were asked to generate their own definition for the term ‘conceptual understanding’. Results indicate that there are some differences among chemistry instructors about the nature of testing conceptual understanding. In particular, depending on what components of conceptual understanding a teacher includes in their definition, items are perceived differently in terms of what they test.

© 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011

Introduction With a seminal publication in 1987, Nurrenburn and Pickering (1) ushered in an era of intense research interest in the apparent disparity between conceptual and quantitative learning among general chemistry students. In the nearly 30 years since this paper was published, dozens of studies have considered similar disparities and often found a number of students who were more capable of answering test items designed to measure quantitative reasoning than conceptual understanding (2–18). A key outcome of this work has been efforts to devise test questions that measure conceptual understanding. While a significant amount of the effort associated with devising conceptual test items has centered on the particulate nature of matter (PNOM) and representations of that level of chemistry (2, 3, 6), there are a number of ways that conceptual understanding has been tested. For example, for topics such as thermodynamic enthalpy change, the use of PNOM diagrams is less central to conceptual understanding (19). An enhanced classification scheme referred to the Expanded Framework for Analyzing General Chemistry Exams (EFAGCE) has been proposed (20) and used in subsequent work (21, 22) to further break down test items in terms of algorithmic, conceptual and recall/definition questions. In the US, one important venue for the development of testing materials is the ACS Exams Institute. There have been several versions of the Conceptual General Chemistry Exam (23–26) produced starting in 1996 which was developed to test conceptual content knowledge as defined by each individual exam committee. In addition, a unique form of exam, called the paired-question exam (27–30), was produced starting in 1997 that specifically addresses topics through both algorithmic and conceptual items. Analysis of the more recent version of the paired-question exam have established psychometric attributes of both the algorithmic and conceptual items (31, 32). In particular, these analyses found that the psychometrics for the conceptual understanding question is not generally lower than the paired algorithmic question. Regardless of progress made in testing related to conceptual learning, there are persistent calls to better identify what defines important conceptual ideas in chemistry, particularly as they relate to general chemistry. Some definitions focus on the cognitive processes required of students. Cracolice (33) has argued that a conceptual problem requires students to use knowledge of a concept behind a problem rather than relying on a memorized recipe for accomplishing an exercise. By contrast, when the first action of a student upon encountering a problem is to decide what recipe (set of mathematical steps) to use, the problem is best defined as algorithmic (34). Recently (35), analysis of over 1400 answers to an open response item in a national survey has given rise to a consensus level definition of conceptual understanding in the minds of chemistry educators. This multipart definition includes earlier published ideas but adds additional features to the articulation of the meaning of conceptual understanding, such that students may demonstrate different facets of the construct. Thus a current “community” definition of conceptual understanding is:

196 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

“A student who demonstrates conceptual understanding can: -

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011

-

-

Apply core chemistry ideas to chemical situations that are novel to the student (Transfer). Reason about core chemistry ideas using skills that go beyond mere rote memorization or algorithmic problem solving (Depth). Translate across scales and representations (Translate). Demonstrate the critical thinking and reasoning involved in solving problems including laboratory measurement (Problem Solve). Expand situational knowledge to predict and/or explain behavior of chemical systems (Predict) (35).”

While this definition is capable of capturing the definitions as articulated by a large majority of the sample in the survey, there are very few participants who included all five components in their open response. Thus, an additional important aspect of understanding the nature of chemistry instructor understanding of student learning about chemical concepts lies in determining how those instructors apply their definitions to the categorization of chemistry test items. This component of the earlier survey work is reported here.

Survey Information The American Chemical Society Examinations Institute (ACS-EI) has conducted survey research work related to assessment for several years. Tests produced by ACS-EI are examples of “grass roots” test development. Each new exam committee determines the most important variables associated with the test to be developed, including the content coverage that the exam will have (36). When the committee that developed the most recent version of the General Chemistry Conceptual Exam (26) first met, in addition to determining chemistry content coverage, they were interested in determining a sense from the chemistry education community about their needs and expectations for the conceptual exam. As a result, in August of 2013, a national survey was conducted that sought to answer several questions related to testing student conceptual understanding. The committee itself created several prototype test items (referred to here as “mock” ACS Exam items) that they expected would provide a range of reactions from educators in terms of whether or not those items test conceptual understanding. While there were additional traits contained in the survey this component is the subject of the current report. Committee members also served as participants for pilot testing of the survey before the full scale data collection was undertaken. Ultimately, the survey contained six sections that participants completed sequentially. They were:

197 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

• • •

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011

• • •

Part 1: Experience teaching general chemistry Part 2: Importance of topics for developing conceptual understanding Part 3: Evaluating whether mock ACS Exam questions are conceptual questions Part 4: Defining ‘conceptual understanding’ Part 5: Student familiarity with visual diagrams and representations Part 6: Course objectives and goals

The material in Part 4 provided the information for prior reports (35) from this survey research, and this paper will focus largely on information from Part 3, where instructors indicated whether or not they believed specific mock exam items test student conceptual understanding.

Survey Data Collection and Participant Demographics Participants were recruited from a regularly maintained database of contact information, including email addresses, for chemistry instructors in the US. ACS-EI does a full-scale revision of this database at least every 2 years. It contains instructors from 2-year colleges, primarily undergraduate institutions and research universities. Roughly 13,000 participants were sent emails requesting their participation in this study. As part of the recruiting process, participants could elect to sign up for random drawings of iPad tablets. Approximately 1800 chemistry instructors logged into the survey system, which was housed within a Survey Monkey account. For the purposes of this study, all who logged into the survey were asked if they had taught general chemistry in the past 5 years and provided the opportunity to give informed consent for their participation. This protocoal was approved as exempt by the Iowa State Institutional Review Board (13-361). A small number of participants did not complete the entire survey. The full sample for analysis included 1519 participants. The data was cleaned to include only the 1296 participants who (a) indicated they had taught general chemistry in the past 5 years; (b) signed the informed consent document and (c) completed most of the items on the survey. The distribution of participants in terms of years teaching is fairly broad as seen in Table 1.

Table 1. Distribution of participants by years teaching Years 0-4 Teaching

5-9

10-14

15-19

20-24

25-29

30-34

35-39

>40

Percent

21.5

18.9

13.7

13.5

7.8

4.5

2.6

3.2

13.0

198 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011

The distribution of participants based on the highest degree offered at their school is summarized in Figure 1 which shows the sample includes instructors from a wide range of environments in general chemistry is taught.

Figure 1. Distribution of school type among participants. Given the variety of institutions from which participants were drawn, it is not surprising that the reported class sizes for general chemistry also varied. Class sizes that were less than 100 students were reported by 71% of the participants. Many of these participants had class sizes less than 50 students, with 45% of the total participants being in this category.

Mock ACS Exam Items This paper reports on the judgements made by participants as to whether or not items designed to mimic those found on ACS Exams test conceptual understanding. A total of six items were devised with the intention of presenting a range of possibilities for participants to consider. Thus, the General Chemistry Conceptual Exam committee suggested some items they collectively felt were likely to be rated as testing conceptual understanding and others that would be less likely to be so rated. This section of the survey followed immediately after queries about the relative importance of various topics in the teaching of general chemistry, and prior to the item asking participants to articulate their definition of conceptual understanding. Results of this latter section have been reported previously (35). The six mock ACS Exam items are provided in Tables 2a and 2b. For each of the 6 mock ACS Exam items, participants were asked to respond on a 5-point likert scale from “Strongly Agree” to “Strongly Disagree”. Specifically, participants were asked to: “Please rate your level of agreement with the statement: If used on an ACS final exam, this question assesses students’ conceptual understanding in General Chemistry.”

199 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011

Table 2a. Mock ACS Exam items 1 through 4 from the survey

200 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011

Table 2b. Mock ACS Exam items 5 and 6 from the survey

Item #1 is about connecting Lewis structures and VSEPR theory and is designated as VSEPR. Item #2 is a LeChâtelier’s Principle question, asking students to understand the direction of response to an external perturbation to an equilibrium system and is designated as “equil”. Item #3 is a form of molar mass calculation, but seeks to heighten the concept of estimation rather than calculator use, and it will be designated as “Molar Mass”. Item #4 is essentially a description of paper chromatography but in this context it is testing the role of intermolecular forces in whether substances dissolve in a particular solvent, and it designated as “IMF”. Item 5 represents what has become a standard means of representing chemistry at the particulate level or a test of student understanding of PNOM and will be designated as “PNOM”. Finally, the expectation of the committee was that item #6 the least likely to be viewed as testing conceptual knowledge. It is a relatively traditional stoichiometry problem and will be designated as “Stoich” here on.

Instructor Impressions of Mock ACS Items As noted earlier, after participants indicated content coverage that they deemed most valuable in general chemistry, they were asked to rate the extent to which the six mock ACS Exam items measure conceptual understanding. The overall responses to this set of items are provided in Table 3. The data at this level reveals that, in the broadest sense, Item 2 (Equil) and 5 (PNOM) are most strongly associated with conceptual understanding. Items 1 (VSEPR) and 4 (IMF) also tilt strongly towards being rated as conceptual. Items 3 (Molar Mass) and 6 (Stoich) have fairly broad views in terms of rating conceptual understanding and neither of these items share the type of consensus seen for items 1, 2, 4 and 5. 201 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011

Table 3. Percentage of faculty rating each Mock ACS Items on a 5-point Likert Scale as measuring students conceptual understanding Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

Strongly Disagree

3.3

1.2

4.2

1.5

2.1

9.0

Disagree

11.2

1.8

20.7

5.1

3.5

23.8

Neutral

13.9

4.8

22.4

10.2

7.7

17.2

Agree

49.3

47.9

35.9

46.5

39.6

26.6

Strongly Agree

22.3

44.4

16.8

36.6

47.1

23.2

Additional insight into how chemistry instructors view the assessment of conceptual understanding is available through the use of the open-response definitions of conceptual understanding. For this analysis, responses have been grouped so that “Strongly Disagree” and “Disagree” are considered together, “Neutral” is considered alone, and “Strongly Agree” and “Agree” are grouped together. As noted in previous work (35), all open response definitions of conceptual understanding have been coded to determine if the participant included any of the five segments of the consensus definition in their response. Some participants include more than one segment, and they are counted in both groups. Thus, it is possible to compare instructors who include a particular idea related to conceptual understanding with all those who do not include that component. Once this grouping was done, statistical analysis using the Mann Whitney U test (37) was performed to determine if any differences are statistically significant between the groups of participants who used each segment and those who did not use the segment in their definition . Because this approach results in multiple comparisons, a post-hoc Bonferoni correction is used to adjust the criterion for significance. This correction means that a p=0.05 significance level requires a p=0.01 value to hold true due to the multiple tests. The results for each item are presented next in a series of stacked-histogram plots with percent of participants in each category: disagree, neutral, agree. There are five groups designated, corresponding to whether or not a definition segment was included in a participant’s definition of conceptual understanding. Finally, the rightmost stacked histogram provides the percentages for the full sample for comparison.

Item 1: VSEPR of One Atom with Two “Central” Atoms The stacked histogram plot for Item 1 is provided in Figure 2. In this case, there is one component of the conceptual definitions where the Mann-Whitney U test show significant difference. Those that express that students be able to translate across scales or representations are less likely to agree that this item is conceptual (p=0.0019).

202 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011

Figure 2. Instructors’ classification of Mock ACS Item 1 (VSEPR) as testing conceptual understanding based on their expressed definitions of ‘conceptual understanding’.

Item 2: Equilibrium and LeChatelier’s Effect The stacked histogram plot for Item 2 is provided in Figure 3. In this case, the Mann-Whitney U test shows no significant differences for any of the different definition segments. This result is unsurprising because a large majority of all participants viewed this items as testing conceptual understanding.

Figure 3. Instructors’ classification of Mock ACS Item 2 (Equil.) as testing conceptual understanding based on their expressed definitions of ‘conceptual understanding’.

203 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Item 3: Molar Mass Calculation, with Ranges for Answers

Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011

The stacked histogram plot for Item 3 is provided in Figure 4. The MannWhitney U test shows significant differences for all of the different definition segments.

Figure 4. Instructors’ classification of Mock ACS Item 3 (Molar Mass) as testing conceptual understanding based on their expressed definitions of ‘conceptual understanding’. Looking at this response pattern in detail, people who include the idea of transferring core chemistry ideas to novel situations are more likely to rank this item as conceptual (p=0.0004). This may imply that the instructors who include the ides of “transfer” in their definition are more likely to expect that having a range available as an answer will be different enough from a customary calculation item that students will perceive a need to do something different. Those who include the idea that conceptual understanding includes depth that goes beyond memorization are less likely to agree that this item tests conceptual understanding (p