[Dissertation] Utilizing Student Data within the Course Management System to Determine Undergraduate Student Academic Success: An Exploratory Study

Table of contents :
Acknowledgments
Table of Contents
List of Tables
List of Figures
List of Formulas
Abstract
1. Introduction
2. Review of Literature
3. Research Methodology
4. Research Results
5. Summary and Discussion
List of References
Appendix A: Records by Course Department
Appendix B: Records by Course Level
Appendix C: Records by Course School
Appendix D: Distribution of Course Grades
Appendix E: Bivariate Correlational Technique for Independent Variables
Vita

Citation preview

UTILIZING STUDENT DATA WITHIN THE COURSE MANAGEMENT SYSTEM TO DETERMINE UNDERGRADUATE STUDENT ACADEMIC SUCCESS: AN EXPLORATORY STUDY

A Dissertation Submitted to the Faculty of Purdue University by John Patrick Campbell

In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy

May 2007 Purdue University West Lafayette, Indiana

ii

To my wife, Lois, Whose love and faith enabled me to think big and made my ongoing pursuit of education possible. and To Matthew and Sarah Campbell, Who helped me keep my perspective on life and waited for me to complete my homework. I love you all.

To David P. Moffett (1956-2004) A friend and colleague who never stopped learning.

iii

ACKNOWLEDGMENTS

I would like to take this opportunity to thank a number of people for their help and support throughout my coursework and dissertation process. Some don’t even know how important they have been, but I would like to acknowledge the following: First I would like to thank the chair of my committee, A.G. Rud, who patiently addressed the numerous questions throughout the process. His willingness to ask the questions and challenge the system made it possible for me to finish this degree program. I particularly appreciate his willingness to intervene when the program was about to fold in order for some students to complete their educational objectives. Thanks also to: Brian French whose expertise and guidance in research methodology was priceless. His enthusiasm for his work is truly contagious. I would never have been able to produce this research without his assistance. He was always patient, always available, and never made me feel stupid - even when I was. Jim Lehman and Bill McInerney for serving on one more committee in addition to all the other students they are mentoring. Their willingness to participate in my educational journey is greatly appreciated. Bart Collins who gave a willing ear and always had time for me – his mentoring and friendship helped me to think more critically and refine my research interests. His knowledge of statistics and ability to take any side of an argument has facilitated the refinement of this work.

iv Mary Moyars-Johnson who edited numerous versions of this document – her expertise was priceless and allowed me to focus on the content rather than the commas. Steve Wanger who taught many of the courses I attended. He provided much of my background in higher education administration. His interest and enthusiasm for the discipline has helped me think more critically and refine my research interests. My colleagues and co-workers at ITaP, especially David Moffett, Mary Moyars-Johnson, Bart Collins, Ed Evans, Gerry McCartney, Jim Bottum, Sharon Steen, Deb Whitten, Carolyn Bogan, and so many others. Each of you has been extremely supportive of me during this endeavor. I could not have asked for better, kinder, or more generous colleagues. My new colleagues at EDUCAUSE, especially Brian Hawkins and Diana Oblinger that helped grow this work into a “Grand Challenge.” My fellow students, from my own cohort and others, who helped and supported each other - Corey Back, Kristie Bishop, Cindy Brewer, Matthew Clawson, Jon Dillow, Stephanie Hopkins, Jolene King, Luke Leftwitch, Rosa Medrano, and others. We have found comfort and comradeship as we each finish the program. We were there for each other through good times and bad times. Paul and Kathy Hanebutt for the use of their summer house as a quiet place to finish my dissertation. Franz Frederick, whose enthusiasm for technology and teaching was truly contagious. Thank you for starting me on this wonderful career. My close friends, Eric, Carole, Katie, and Aimee Hahm, your friendship has been a source of stability and comfort. To my bible study group including the Berryman’s,

v Buess’, Hahm’s, Hecht’s, Patterson’s, Reeder’s, and the Wegener’s – thank you for helping me balance my school life and my spiritual life. Most importantly, I need to thank my family. Lois, my wife, has actively supported my decision to obtain a PhD and has provided the stability and encouragement I needed to succeed. With the endless nights away from home or doing homework, I could not have accomplished this without you. My children, Matthew and Sarah, have been wonderful in providing me encouragement, love, and patience. Even during this difficult time, you made sure I experienced more in life than school and work. My mother, June, who patiently kept asking – “when are you going to finish school?” I am now finished mom!

vi

TABLE OF CONTENTS

Page LIST OF TABLES............................................................................................................ xii LIST OF FIGURES .......................................................................................................... xv LIST OF FORMULAS .................................................................................................... xvi ABSTRACT.................................................................................................................... xvii CHAPTER 1: INTRODUCTION ....................................................................................... 1 Student Persistence ........................................................................................................1 Grades and Academic Success.......................................................................................2 Technology and Predicting Academic Success .............................................................3 Purpose of the Study ......................................................................................................4 Research Questions Specified........................................................................................5 Significance of this Study ..............................................................................................5 Importance to the Institution....................................................................................7 Importance to the Student ........................................................................................8 Importance to Society ..............................................................................................8 Summary ......................................................................................................................10 Organization.................................................................................................................10 CHAPTER 2: REVIEW OF LITERATURE.................................................................... 11 Retention in Higher Education.....................................................................................11 Retention Models .........................................................................................................12 Tinto’s Theory of Student Departure.....................................................................13 Astin’s Theory of Student Involvement.................................................................14 Justification for Retention Planning.............................................................................17 College Preparation......................................................................................................17

vii Page Standardized Aptitude Testing...............................................................................18 High School Preparation ........................................................................................19 Predictive College Variables........................................................................................20 College Grade Point Average ................................................................................20 Role of Faculty in Retention ..................................................................................21 Support from Other Campus Activities .................................................................22 Student Effort.........................................................................................................22 Other Factors Influencing Academic Success and Retention ......................................23 Use and Growth of Course Management Systems ......................................................24 Faculty Adoption of Course Management Systems...............................................26 Tool Usage Within Course Management Systems ................................................27 Summary ......................................................................................................................29 CHAPTER 3: RESEARCH METHODOLOGY .............................................................. 31 Research Design...........................................................................................................31 Participants...................................................................................................................31 Development of Data Subgroups .................................................................................32 Protection of Human Subjects .....................................................................................35 Independent Variables .................................................................................................36 Dependent Variables....................................................................................................41 Preparation of Data for Analysis..................................................................................41 Removal of Extraneous Records............................................................................42 Treatment of Missing Data ....................................................................................42 Calculated Variables ..............................................................................................46 Coding of Independent Variables for Analysis......................................................47 Standardization of Variables..................................................................................48 Removal of Outliers...............................................................................................49 Research question 1: Correlational analysis ................................................................49 Research question 2: Reduction of Variables ..............................................................50 Reduction through Affiliation................................................................................50

viii Page Reduction through Factor Analysis .......................................................................51 Assumptions of Factor Analysis ......................................................................52 Sample Size................................................................................................52 Linearity.....................................................................................................52 Absence of Multicollinearity .....................................................................52 Factorability of R .......................................................................................53 Extraction of the Factors..................................................................................53 Research Question 3: Predicting Student Success .......................................................53 Assumptions of Logistic Regression .....................................................................54 Ratio of Cases to Variables....................................................................................54 Adequacy of Expected Frequencies.......................................................................55 Absence of Multicollinearity .................................................................................55 Independence of Errors ..........................................................................................55 Significance Tests for Logistic Regression............................................................56 Likelihood Ratio ..............................................................................................56 Chi-square Test of Goodness-of-Fit.................................................................56 Effect Size........................................................................................................57 Research Question 4: Predictive Power of the Model .................................................57 Reliability.....................................................................................................................58 Validity ........................................................................................................................58 Hypothesis Testing – Type I and Type II Error...........................................................59 Limitations of Study ....................................................................................................60 Summary ......................................................................................................................61 CHAPTER 4: RESEARCH RESULTS ............................................................................ 63 Characteristics of Dependent Variables.......................................................................63 Research Question 1: Correlation Between CMS Factors and Academic Success .....70 Research Question 2: Reduction in Independent Variables.........................................73 Research Question 3: Utilizing the Independent Variables to Predict Success ...........79 Main Regression Model.........................................................................................79

ix Page Assumption for Regression..............................................................................79 Results of the Binary Regression Analysis......................................................81 Reduced Main Model.............................................................................................89 Model with CMS Usage.........................................................................................93 Main Model Comparison Between Freshmen and Non-Freshmen Students.......104 Reduced Main Model – Freshmen Students vs. Non-Freshmen Students...........111 Predictive Power of the Model ..................................................................................114 Independence of Samples ....................................................................................114 Model Validity – Reduced Main Model ..............................................................117 Model Validity – Reduced freshmen student model............................................119 Conclusion .................................................................................................................120 CHAPTER 5: SUMMARY AND DISCUSSION .......................................................... 122 Summary of the Results .............................................................................................122 Correlation Between CMS Factors and Academic Success ................................122 Predicting Academic Success Using the CMS ....................................................123 Utilizing the CMS and Demographic Data to Predict Academic Success...........124 Predictive Power of the Model ............................................................................125 Further Considerations of the Research...............................................................125 Implications for Practice ............................................................................................126 Development of an Academic Retention Data Warehouse..................................126 Development of an “Academic Effort Scale”......................................................129 Awareness ............................................................................................................130 Student Awareness.........................................................................................130 Faculty and Instructor Awareness..................................................................131 Advisor Awareness ........................................................................................131 Notification Systems......................................................................................132 Assistance ............................................................................................................133 Engagement..........................................................................................................134 Program Development .........................................................................................134

x Page Early Adopters .....................................................................................................135 Implications for Practice: Concluding Remarks ..................................................136 Implications for Research ..........................................................................................137 Relationship to Previous Research.......................................................................137 Suggestions for Additional Research...................................................................138 Additional Academic Data.............................................................................138 Step by Step Analysis ....................................................................................139 Exploring Additional Populations..................................................................140 Additional Research: Concluding Remarks.........................................................140 The Obligation of “Knowing” ...................................................................................140 Ethics and the Ethics of Teaching........................................................................142 Raising the Ethical Question: The Power of Knowing........................................146 Examining the Ethical Dilemma: The Potter’s Box ............................................147 Definition / Facts............................................................................................148 Values ............................................................................................................149 Principles........................................................................................................149 Loyalties.........................................................................................................150 Faculty Member, Student, and Institutional Responsibilities ..............................150 Faculty Member’s Responsibilities................................................................151 Student’s Responsibilities..............................................................................151 Institution’s Responsibilities..........................................................................151 Implications of Ethical Examination ...................................................................152 Recommendations................................................................................................152 Obligation of Knowing: Concluding Remarks ....................................................155 Conclusion .................................................................................................................155 LIST OF REFERENCES................................................................................................ 157 APPENDICES Appendix A: Records by Course Department ...........................................................170 Appendix B: Records by Course Grade.....................................................................171

xi Page Appendix C: Records by Course School ...................................................................172 Appendix D: Distribution of Course Grade ...............................................................173 Appendix E: Bivariate Correlational Technique for Independent Variables.............174 VITA ............................................................................................................................... 178

xii

LIST OF TABLES

Table

Page

Table 1.1: Benefits of higher education...............................................................................9 Table 3.1: Student demographics.......................................................................................32 Table 3.2: Categorical student demographics....................................................................32 Table 3.3: Model development subgroup - Student demographics ...................................33 Table 3.4: Model Development subgroup - Categorical student demographics................34 Table 3.5: Model Verification Subgroup - Student demographics ....................................35 Table 3.6: Model Verification Subgroup - Categorical student demographics .................35 Table 3.7: Course management system independent variables .........................................36 Table 3.8: Demographic and cognitive independent variables..........................................39 Table 3.9: Summary of missing cases................................................................................44 Table 3.10: Calculated independent variables ...................................................................46 Table 3.11 Coding of independent variables .....................................................................47 Table 3.12: Bivariate correlational technique....................................................................50 Table 3.13: Course management system composite variables...........................................51 Table 4.1: Number of students each course grades............................................................64 Table 4.2: Distribution of course grades by ethnic background ........................................65 Table 4.3: Distribution of course grades by gender...........................................................66 Table 4.4: Distribution of course grades by course level...................................................67 Table 4.5: Distribution of course grades by course school................................................69 Table 4.6: Correlations between CMS variables ...............................................................72 Table 4.7: Total variance explained by components (oblique rotation) ............................74 Table 4.8: Comparison of eigenvalues...............................................................................76 Table 4.9: Rotated component pattern matrix....................................................................77

xiii Table

Page

Table 4.10: Rotated component structure matrix...............................................................77 Table 4.11: Component correlation matrix........................................................................78 Table 4.12: Main Model Collinearity Diagnostics ............................................................80 Table 4.13: Results of the Hosmer-Lemeshow Test..........................................................81 Table 4.14: Regression results for the main model............................................................83 Table 4.15: Block by block regression results for the main model ...................................84 Table 4.16: Block by block improvement of χ2 for the main model..................................88 Table 4.17: Reduced main model ......................................................................................90 Table 4.18: Block by block regression results for the reduced main model......................91 Table 4.19: Comparison of regression results – main model vs. reduced model ..............93 Table 4.20: Summary of course use of the course management system ...........................94 Table 4.21: Comparison of predictive power of models by CMS usage ...........................94 Table 4.22: Comparison of variance accounted by CMS usage ........................................95 Table 4.23: Main Model –Summary of significant predictors by CMS usage ..................97 Table 4.24: Main model – fourth quartile (highest)...........................................................98 Table 4.25: Main model – third quartile ............................................................................99 Table 4.26: Main model – second quartile.......................................................................100 Table 4.27: Main model – first quartile (lowest) .............................................................101 Table 4.28: Summary of classification rates....................................................................102 Table 4.29: Distribution of course grade by classification ..............................................104 Table 4.30: Summary of freshmen student vs. non-freshmen student models ................105 Table 4.31: Comparison of R2 between freshmen and non-freshmen models .................105 Table 4.32: Main model –summary of significant predictors by classification...............107 Table 4.33: Main model – non-freshmen students...........................................................108 Table 4.34: Main model – freshmen students..................................................................109 Table 4.35: Summary of classification rates....................................................................110 Table 4.36: Reduced main model – non-freshmen students ............................................112 Table 4.37: Reduced main model –freshmen students ....................................................112 Table 4.38: Summary of freshmen student vs. non-freshmen student reduced models ..113

xiv Table

Page

Table 4.39: Variability within the freshmen vs. non-freshmen student model................113 Table 4.40: Summary of Classification Rates..................................................................113 Table 4.41: Comparison of independent samples ............................................................114 Table 4.42: Comparison of independent samples: t-test..................................................116 Table 4.43: Comparison of predicted and actual success ................................................118 Table 4.44: Comparison of predicted and actual success ................................................118 Table 4.45: Comparison of predicted and actual success in reduced freshmen model....119 Table 4.46: Comparison of predicted and actual success in reduced freshmen model....119 Table A.1: Records by course department.......................................................................170 Table A.2: Records by course level .................................................................................172 Table A.3: Records by course school ..............................................................................173 Table A.4: Distribution of course grades.........................................................................174 Table A.5: Bivariate correlational technique for independent variables .........................175

xv

LIST OF FIGURES

Figure

Page

Figure 2.1: Tinto’s longitudinal model of institutional departure (simplified)..................15 Figure 3.1: The issue of cause and effect...........................................................................60 Figure 4.1: Number of students enrolled by course level ..................................................66 Figure 4.2: Mean student grade by course level ................................................................67 Figure 4.3: Mean student grade for non-College of Science courses ................................68 Figure 4.4: Mean student grade for College of Science courses........................................68 Figure 4.5: Percentage of failing grades by home college.................................................70 Figure 4.6: Scree plot of component eigenvalues..............................................................75 Figure 4.7: Correct classification of the main model.........................................................83 Figure 4.8: Correct classification of the reduced model ....................................................92 Figure 4.9: Nagelkerke R2 by CMS usage .........................................................................96 Figure 4.10: Nagelkerke R2 by CMS usage .....................................................................103 Figure 4.11: Change R2 between freshmen and non-freshmen models ...........................106 Figure 4.12: Classification rate by model ........................................................................110 Figure 4.13: Correct classification of students needing help – reduced model ...............118 Figure 4.14: Correct classification of students needing help – freshmen model .............120 Figure 5.1: Academic retention data warehouse architecture..........................................129

xvi

LIST OF FORMULAS

Formula

Page

Formula 3.1: Calculating a Z score....................................................................................49 Formula 3.2: Logistic regression equation.........................................................................54

xvii

ABSTRACT

Campbell, John Patrick, Purdue University, May, 2007. Utilizing Student Data within the Course Management System to Determine Undergraduate Student Academic Success: An Exploratory Study. Major Professor: A.G. Rud. For nearly six decades, researchers have been studying the issues of student persistence and retention in higher education. Despite the decades of research and projects to improve retention, overall retention figures have remained between 45% and 50%. With the contentious debates over the Higher Education Reauthorization Act in 2005 and increasingly constrained federal and state budgets, the demand for increased accountability from the students, families, and the legislature has required higher education institutions to renew their focus on issues of academic quality, cost effectiveness, student retention, and graduation rates. This research expands traditional retention and academic success studies by introducing additional student behavioral data from the course management system (CMS). By examining more than 70,000 records from the CMS, the researcher sought to determine which information, if any, was the strongest indicators of student success. The initial portion of this study reduced the twenty CMS variables into five factors. Subsequent portions of the study utilized regression techniques to identify the key variables necessary to predict academic success. The reduced main model accurately predicted nearly sixty-six percent of the students needing help. A more focused freshmen-only model was able to accurately predict nearly eighty percent of the students needing help. All of the models were validated through the use of an additional sample. While much work remains in the use of analytics to predict student success, this study provides an initial validation that institutions can become more effective by utilizing the course management system as a predictor of academic success. Through the

xviii use of academic data, students, faculty members, and academic advisors can be better equipped with additional information to improve the effectiveness of their portion of the academic environment. While additional work remains to develop this research study into a scalable, institutional solution, the framework has been set for utilizing academic information to improve the institution’s academic success and retention efforts. Considering the growing interest in retention within higher education, it is hoped that additional research will conducted to enhance the model within this study, providing academic decision-makers access to a more comprehensive set of tools and information.

1

CHAPTER 1. INTRODUCTION

For nearly six decades, researchers in higher education have been studying the phenomena of undergraduate student persistence, retention, and academic success. The study of student persistence and retention began with the seminal work of Vincent Tinto (1975) and others (Astin, 1975) on the examination of student dropout characteristics. Since that time, the research has transformed from the study of dropout characteristics to the development of holistic models of attrition and retention from the perspective of student-institution interaction. As a result of the extensive literature and discussion on the topic, retention rates have become a key indicator of overall effectiveness at many institutions (Astin, Korn, & Green, 1987). The interest in student persistence and retention varies from institution to institution, but can be frequently traced to the growing federal demand for accountability in higher education as demonstrated by the contentious debate of the 2005 Higher Education Reauthorization Act. For public institutions, the federal debate over accountability is frequently mirrored, if not intensified, during the creation of state budgets since legislators frequently question why the cost of higher education has outpaced inflation and other economic indicators. Demands to increase the effectiveness and efficiency of higher education have put pressure on institutions to improve retention and graduation rates. In turn, these pressures have forced colleges and universities to examine student academic success and the educational and support programs of the institutions themselves.

1.1. Student Persistence One of the first attempts to explain student persistence was made by Alexander Astin, who created the Input-Environment-Outcome model to serve as a conceptual

2 framework for studying student persistence. The purpose of the model was “to assess the impact of various environmental experiences by determining whether students grow or change differently under varying environmental conditions” (Astin, 1993, p. 7). Astin argued that one must examine the preexisting characteristics prior to entering college (inputs), the environmental factors of the institution (size, student involvement, etc.), and the effects of college (outcomes). Astin (1991) identified 146 possible input (precollege) variables including: age, race, high school grades, and reasons for attending college. Astin (1991) also identified 192 environmental variables which might influence student success. These were broken into eight classifications: institutional characteristics, students’ peer group characteristics, faculty characteristics, curriculum, financial aid, major field of choice, place of residence, and student involvement. The final component of Astin’s model was outcomes. Astin identified 82 outcomes including academic cognition, career development, and retention. Tinto (1975) built upon Astin’s efforts to explain the variables that influence student persistence. Tinto theorized that students enter with a certain set of characteristics that increase or decrease their commitment to and integration into the institution. Tinto stated that greater integration leads to higher retention while Pascarella and Terenzini (1991) furthered this notion by stating that “negative interactions and experiences tend to reduce integration” (p. 53) which ultimately leads to withdrawal. While studies and models for student persistence vary in many ways, the basic notion remains the same – persistence and retention are influenced by a number of complex factors, and the responsibility for improving persistence is a shared responsibility between the institution and the student.

1.2. Grades and Academic Success The first step in any student’s college career is applying for admission to the institution. In the admissions process, institutions utilize a number of the characteristics identified by Astin, Tinto, and others to make decisions about which students are likely to succeed within their institutional environment. The institutional admissions process attempts to predict which students are “likely to succeed academically” if admitted and

3 which students will “contribute academically and non-academically” to the goals of the institution. Once admitted, both the student and institution share mutual responsibility for the persistence, retention, and ultimate academic success of the individual. Academic success can be measured in a variety of ways including the demonstration of learning outcomes, persistence in study from academic term to academic term, college grade point average, graduation rate, and employment placement. Of the measures of academic success, probably no other variable has been used more frequently to describe academic success or has attracted more attention than grade performance (Pascarella & Terenzini, 2005). Astin (1993) notes that “grades are hardly a perfect measure of learning and intellectual development in that they generally reflect a student’s performance relative to other students rather than how much has been learned.” From a research perspective, grades are also troublesome since the calculations of grades and the standard by which they are applied vary enormously both within and across faculty, academic departments, and institutions. Despite the limitations and flaws of utilizing grades as a measure and predictor of academic success, grades remain the single best predictor and largest contributor to student persistence, degree completion and graduate school enrollment (DesJardins, Ahlburg, & McCall, 1999; Pascarella & Terenzini, 2005). In addition, grades remain the most consistent predictors of these outcomes in large multiinstitutional studies as well as in single institution studies.

1.3. Technology and Predicting Academic Success Colleges and universities have access to enormous amounts of data about their applicants and have developed very sophisticated predictive admission formulas. However, most institutions have not taken advantage of the enormous volume of data amassed by the institution itself to contribute to the ongoing analysis and prediction of student success beyond the first year. More importantly, institutions have failed to leverage the data from course management systems, electronic grade books, electronic

4 portfolio systems, and similar academic tools to enhance their prediction of ongoing student academic success. With the current state of computing hardware and software, as well as the maturing of enterprise academic software, institutions can take advantage of the increased access to data to develop new analytical techniques to predict academic success. Course management systems (CMS) alone play an increasingly important role as a core instructional resource on many campuses. Green (2005) reports a substantial increase in course management system usage from 14.7% in 2000 to 45.9% in 2005. As a result, 52.4% of institutions surveyed indicate that they have a strategic plan for the implementation and growth of a course management system. Since course management systems have matured to encompass extensive databases that track many aspects of student performance and behavior, methodologies can be developed to use this data to provide an ongoing predictive model for academic success. As a result, retention models which were once prohibitively expensive, complex, or distributed can now be developed into an intelligent early warning system. Given the wide deployment of course management systems, higher education could greatly benefit from exploration of these systems as part of a larger retention system. Therefore, this study evaluated the usefulness of course management system data in combination with other student information in predicting student outcomes.

1.4. Purpose of the Study The purpose of this study was to develop a model utilizing a combination of course management tracking data and demographic variables to accurately predict academic success of undergraduate students at a large, public, research university located in the Midwest. If specific variables can be identified as predictors of success, or specific predictors can be found which indicate potential problems, early intervention and types of assistance can be designed and implemented. The end result will be more undergraduates who are academically successful, persist in college, and are satisfied with their experience.

5 1.5. Research Questions Specified The following research questions provided the focus for this study: •

R1: Are there data elements in the WebCT Vista course management system which can be correlated with student success in undergraduate courses at a large, research university? (Correlation)

•

R2: Are there data elements in the WebCT Vista course management system which can be combined to predict student success in undergraduate courses at a large, research university? (Factor Analysis)

•

R3: How do the data elements in the WebCT Vista course management system and in enrollment management combine to predict student success in undergraduate courses at a large, research university? (Hierarchical Regression)

•

R4: What is the predictive power of the proposed model using course management system and demographic data to predict student success in undergraduate courses at a large, research university? (Use different subsamples for model development and testing. Compare classification tables)

1.6. Significance of this Study Students, families, and legislatures have become increasingly concerned about the rising costs of tuition and are demanding increased accountability from colleges and universities. These constituencies are demanding more cost effective and efficient ways of accessing and attaining postsecondary education. Magrath (1996) in his report to the Kellogg Foundation entitled The Challenges and Opportunities for Land-Grant Universities in the Twenty-First Century states that universities “must recognize the new realities of diminished public resources while forthrightly facing [their] shortcomings” (p. 18). Magrath continues with the primary challenges which universities need to address: Clearly, these include our need to use faculty time more productively, our obligation to pay more attention to undergraduate students and to become full-time collaborators with public schools, and our duty to link research discoveries and educational insights with our states and communities in partnerships that strengthen our economy and society. And we dare not be

6 afraid to use new technologies – most of them spawned in our universitiesto improve how we teach, learn, and communicate in a world not defined by campus boundaries or restricted by towers built of ivory (p. 17-18). Due to a decline in student satisfaction and the general erosion of public confidence, colleges and universities “have been challenged by their various constituents to demonstrate student success” (Saunders & Burton, 1996, p. 555). In order to best meet the challenges of accountability from a “consumer satisfaction” perspective, many colleges and universities have chosen to measure and demonstrate their successes in terms of retention and graduation rates. Using retention and graduation rates as an indication of productivity and outcomes, universities attempt to speak to students, community members, legislatures, and other constituents in a way which directly addresses public concerns. By demonstrating positive retention rates and graduation outcomes, colleges and universities hope to assuage rising consumer anxiety and persistent questions concerning the operational efficiency of their institutions and the quality of the undergraduate education they offer. Despite the use of retention to demonstrate institutional quality and the extensive efforts to admit students who will succeed, the degree completion rate is alarmingly low. Tinto (1993) suggests that “more students leave their college or university prior to degree completion than stay” (p.1). A survey conducted by the American College Testing (ACT) Program (2004) found that 30 percent of all students entering a four-year public institution leave before the beginning of their second year. Of postsecondary students entering four-year institutions in pursuit of a bachelor’s degree in 1995-96, only 57 percent had earned a bachelor’s degree within six years (National Center for Education Statistics, 2004). Although the attrition/retention/persistence phenomenon has been exhaustively researched since the 1960’s, college student attrition has consistently remained at approximately 45-50 percent nationally since the turn of the century (Tinto, 1993). At Purdue University, the retention rate has held steady at 86% while the six year graduation rate is 72% (Purdue University, 2006). As we look to the future, questions must be asked: Why continue to invest fiscal and human resources studying retention? Why place the focus on retaining students? As one examines the larger landscape which takes into

7 account the institution, the student, and society, there are several important reasons to examine retention.

1.6.1. Importance to the Institution The economic impact of retaining college students is generally the first and most easily understood reason advanced by institutions of higher education. The concept is rooted in mathematics and the equation is simple: students equal tuition dollars (Bean, 1986). Speaking in broad terms, American higher education consists of a vast and complex system of public and private institutions which include two-year and four-year colleges as well as multiple degree-granting research universities. Viewed through the lens of corporate culture, higher education is a 132 billion dollar-per-year enterprise (state and federal funds only) which serves more than 16.6 million students annually (Chronicle of Higher Education, 2005). Further, the higher education enterprise employs approximately 3.2 million people (Chronicle of Higher Education, 2005). Putting these substantial numbers into perspective, the United States has a sizable fiscal and human resource investment in higher education. Retention rates are often regarded as important indicators of institutional quality and commitment to undergraduate education. Institutional retention rates impact public perception, institutional reputation, future enrollments, and faculty morale. “When faculty teach at an institution where attrition is high, they are likely to feel negative toward themselves and their profession” (Bean, 1986, p. 48). This negative morale then extends beyond the faculty to the entire university community. The reputation and status of the college or university declines and the campus culture becomes disaffected. “Attrition is ordinarily viewed as student failure, but institutions with high rates of attrition can also be viewed as failures, and the best students, faculty, staff, and administrators will try to leave” (Bean, 1986, p. 48). This supports the fundamental need to retain college students and continue retention related programming and research.

8 1.6.2. Importance to the Student One role of higher education is to help an individual develop, to the limits of his or her capability, the ability to understand and examine issues on an ethical, academic, and practical level. Further, higher education provides the individual with an increased opportunity to gain meaningful and rewarding employment. According to a 2002 report from the US Census Bureau, a college graduate will earn a yearly average of 1.8 times the amount of someone with only a high school diploma (Day & Newberger, 2002). In addition, any student who has at least some education past the high school diploma can be expected on average to earn three hundred thousand dollars more over his or her lifetime than those who do not. The data indicates that the demand for skilled employees is high; to prospective employers a college graduate is worth almost twice as much as a high school graduate. As the better jobs within the economy continue to evolve towards technological and knowledge-based industries, one can only assume that at least some college education will be necessary to function within that economy. Post-secondary education will become a necessary means to viability in the labor market and will no longer be linked only with earning a place in the upper class.

1.6.3. Importance to Society A college education contributes significantly to the public’s shared economic, social, and cultural well-being. While the benefit of a larger lifetime income is the frequent focus of the private individual, we must consider the economic and social benefits to the society as a whole. These benefits include: •

increased tax revenues

•

decreased reliance on public assistance programs

•

lower unemployment rates

•

increased voting, volunteering, and other civic activities For example, the unemployment rate in January 2004 of those without a high

school diploma was 8.8%. This compares to unemployment of 4.9% for those with a

9 high school diploma and only 2.9% for those with a college degree. (Institute for Higher Education Policy and Scholarship America, 2004). In the end, successful economies depend on one or more of three factors: natural resources, cheap labor, or a highly educated workforce. There is no doubt that the future of the United States’ information economy significantly depends upon the third factor. The contribution that higher education can make to the economy is intimately connected to the maintenance and enhancement of learning and inquiry. The equations are these: “No learning = No skills; No higher learning = No higher skills; No inquiry = No invention” (University of Edinburgh, n.d.). This concept of the varying contexts and benefits of higher education can be best visualized through the following table provided by the Institute for Higher Education Policy (2004, p.8):

Table 1.1: Benefits of higher education

Economic

Social

Public • Increased tax revenues • Greater productivity • Increased consumption • Increased workforce flexibility • Decreased reliance on government • Reduced crime rates • Increased charitable giving • Expanded community service • Increased quality of civic life • Appreciation of diversity • Improved ability to adapt to and use technology

Private • Higher salaries and benefits • Increased employment • Higher savings levels • Improved working conditions • Increased personal and professional mobility • Improved health/life expectancy • Improved quality of life for offspring • Better consumer decision making • Increased personal status • More hobbies, leisure activities

10 1.7. Summary Public priorities will continue to include improving access, graduation rates, and learning. As institutions provide greater access to higher education, they can expect an accompanying increase in problems pertaining to the persistence of students in all academic programs. In academic year 1996-1997, approximately 16.6 million students participated in associate, bachelor, masters, doctoral, or professional degree programs. During this same period, the 6-year graduation rates at 4-year institutions was 54.4% (Chronicle of Higher Education, 2005).

The increased demand for accountability

requires substantial institutional improvements in academic quality, and cost effectiveness which can be gained through better utilization of available data. Toward this end, the utilization of course management data to predict ongoing academic success constitutes a promising step towards increasing sophistication of models and theories. The course management system provides “real time” information to the student, faculty member, and the institution which may be utilized to extend current models and become the foundation for future retention efforts.

1.8. Organization This dissertation is an exploratory study to identify the combination of demographic information and behavioral factors found within the course management data that may be useful as predictors to student success. Chapter 2 consists of a review of the literature and theories related to student retention and academic success. In Chapter 3, the methodological framework, as well as the data collection and analysis procedures are provided. Chapter 4 provides the analysis of 71,794 individual records to determine which factors, if any, are the strongest indicators of student success within courses utilizing CMS. Finally, in Chapter 5, the researcher summarizes the study, draws conclusions, and offers recommendations for practice and implications for future research.

11

CHAPTER 2. REVIEW OF LITERATURE

The ability to predict student academic performance and retention within higher education has long been a focus of faculty and institutional research. The results of this research have led to the development of admissions formulas, descriptive models, assessment tools, and numerous journal articles. Despite the wealth of research, institutional practices based on the educational research are frequently limited to the admissions process and selective first year retention efforts. As national retention rates remain steady near fifty percent, institutions struggle to develop long-term strategies to ensure student academic success and retention. This chapter provides an overview of the broad issues and trends which influence student retention. The literature review presented below includes sections which outline the basic theories of retention, provide an overview of pre-college predictors of academic success and retention, and discuss how faculty and student behaviors influence retention. The final section of this chapter reviews the current status of course management systems within higher education.

2.1. Retention in Higher Education Accurately predicting academic success and retention has been examined from a number of different directions (Angoff, 1982; Baron & Norman, 1992; Sedlacek, 1989). One of the earliest applications for predicting academic success was the college admissions process. As the demand for higher education grew, institutions turned to various mathematical models based on high school records and standardized examinations. Since the 1960’s, the focus has expanded to include retaining students who had been admitted to the institution.

12 In an attempt to develop a predictive model of student success and retention, researchers have approached the problem in a number of ways including the match between the institution and student (Tinto, 1975) and the level of student involvement (Astin, 1977, 1993). Based on the research of Tinto, Astin, Pascarella, Bean, and others, numerous individual, environmental, and institutional characteristics have been identified as being effective predictors of student academic performance and retention within higher education. The theories of student retention by Tinto and Astin outlined below will provide a framework for this research. While this study does not focus specifically on retention, academic performance is considered an important part of many of the student retention models. If this research is proven to provide accurate predictors of success, the theories will also provide a potential intervention strategy to improve student retention.

2.2. Retention Models Pascarella and Terenzini (1991) describe the retention theories within this section as “college impact models of student change.” These theories assign a “prominent and specific role to the context in which the student acts and thinks” (Pascarella & Terenzini, 1991, p. 57). [Institutional] structures, policies, programs, and services (whether academic or nonacademic), as well as attitudes, values, and behaviors of the people who occupy (and to some extent define) institutional environments, are all seen as potential sources of influence on students’ cognitive and affective changes” (Pascarella & Terenzini, 1991, p. 57). As Pascarella and Terenzini (1991) discuss in their definitive work How College Affects Students, college impact models consider the university environment “as an active force that not only affords opportunities for change-inducing encounters but can also, on occasion, require a student to respond [to change]” (p. 57). While other factors also may have a role in the individual students’ decision to remain in school, the institutional environment plays a significant role in the students’ development and ability to achieve their educational goals.

13 The section below reviews two major retention theorists that have contributed to the body of literature regarding college student retention.

2.2.1. Tinto’s Theory of Student Departure The first, and perhaps most citied, college impact model of student retention was developed by Vincent Tinto. In 1975, Tinto developed a conceptual model of student retention wherein he argued that students remain at an institution depending upon how well they perceive themselves to be integrated into the institutions’ academic and social system. Tinto provided a more explicit, longitudinal, and interactive model (Figure 2.1). Tinto (1993) described his model as an “interactive model of student departure” (p. 113). In this model, students enter higher education with a diversity of personal, family, and academic characteristics and skills. Collectively, these student characteristics and skills lead to the formation of an initial commitment to the institution and to their personal educational goals. The initial commitment influences how well the student becomes integrated into the social and academic fabric of the institution. Over time, the student’s subsequent interactions and experiences within the institutional environment influence the student’s overall perceptions about their fit at college. Positive interactions within the environment presumably lead to greater student integration. As integration increases, the student’s commitment to their personal educational goals and participation within the institution increases. Likewise negative interactions impede integration into the social and academic environment and ultimately lead to disengagement from the institutional academic and social environment. As the disengagement continues, a student decides to leave an institution when he or she no longer perceives a match with the institutional environment (Pascarella & Terenzini, 1983). Tinto (1975) summarized that persistence or failure in college could be examined in terms of: A longitudinal process of interactions between the individual and the academic and social systems of the college during which a person’s experiences in those systems… continually modify his goal and institutional commitments in the ways which lead to persistence and/or varying forms of dropout. (p. 94)

14 Positive social and academic experiences increase a student’s institutional and academic goal commitments, thus leading to an increased likelihood for persistence. Within Tinto’s model, academic integration was viewed in terms of grade performance and intellectual development. Social integration was related to peer-group and faculty interactions. Tinto’s research demonstrates the importance of academic success as it relates to student retention. Tinto’s research also provides a solid reason to increase faculty or student faculty interactions based on the results of any student academic success and retention model.

2.2.2. Astin’s Theory of Student Involvement One of the earliest college impact retention models stems from the work of Alexander Astin (Pascarella & Terenzini, 2005). Astin’s (1975, 1984) theory of student involvement begins with his “input-environment-outcome” model. Astin’s model (1993) examines the impact of student interactions with faculty and peers as the key to postsecondary retention.

15

Pre-entry characteristics

Intentions, goals, and commitments

Academic experiences

Social experiences

Integration

Intentions, goals, and commitments

Departure decision Figure 2.1: Tinto’s longitudinal model of institutional departure (simplified)

16

The fundamental elements of the model are defined in the following manner (Astin, 1993): Inputs refer to the characteristics of the student at the time of initial entry to the institution; environment refers to the various programs, policies, faculty, peers, and educational experiences to which the student is exposed; and outcomes refers to the student’s characteristics after exposure to the environment (p. 7, emphasis in original text). Astin’s theory of involvement is basic in its foundation – “students learn by becoming involved” (Astin, 1985, p. 133). The more students are involvement in the academic and social aspects of college, the more they learn. An involved student devotes considerable time and energy to academics, spends time on campus, participates in extracurricular activities, and frequently interacts with faculty. While, the degree of involvement is determined by the student, “learning, academic performance, and retention are positively associated with academic involvement, involvement with faculty, and involvement with student peer groups” (Astin, 1993, p. 394). The learning and development that occur are influenced by the quality and quantity of student interaction (Astin, 1975). High quality interaction requires an investment of energy by the student in academic activities. Frequent interaction with faculty is more strongly related to satisfaction with college than any other type of involvement. Activities which require students to become more active participants in their academic experiences improved college student retention and academic performance (Astin, 1984). Like Tinto, Astin recognized the importance of the institutional environment, but Astin emphasized the student’s role in capitalizing on the opportunities made available by the environment. The critical components that improved college student retention and academic performance were related to factors that facilitated students becoming active participants in their experience, as opposed to passive recipients of knowledge. The amount of time and effort invested by the student varies depending on his/her goals, interests, and commitment (Astin, 1984).

17 As a result of Astin’s theory of involvement, several important practical applications have evolved for faculty and institutions. One primary application is the need to change the classroom experience from the delivery of course content, to emphasizing participation in the learning process. The change requires faculty members to become aware of the student motivations and develop techniques for increasing student involvement in the learning process (Astin, 1993). In the end, the institutional efforts to promote retention can be judged by the amount of student interaction they foster. Astin argues that if faculty, staff, and administrators unify their energies to promote student involvement, students will likely become stronger learners and more likely to persist through graduation (Astin, 1993). Astin’s research provides a basis for the importance of this current study in promoting student awareness of their involvement within the course and/or curriculum.

2.3. Justification for Retention Planning College student attrition has remained steady between forty-five and fifty percent (45-50%) since the turn of the century (Tinto, 1993). Since the 1960’s, the attrition/retention phenomenon has been exhaustively researched, delineated, and presented in a variety of methods designed to enhance our understanding of the reason for students leaving without completing their work and graduation. Perhaps the most compelling reason to move forward with any retention research is the institutional obligation to support, educate, and provide guidance to each student it admits (Bean, 1986). With that ethical obligation in mind and after nearly five decades of research, new sources of data and analysis methods may provide exciting avenues for additional research.

2.4. College Preparation Higher education institutions have utilized the wealth of academic success and retention research to identify predictive variables that fit the institution’s environmental context. Based on this research, each institution annually adjusts its admissions formulas

18 to meet particular institutional goals and desired outcomes. Within the institution, the admission’s formula is adjusted regularly to remain responsive to federal, state, and community needs which it serves. College administrators have routinely used predictive admissions formulas that include standardized test scores, high school grade point averages, and class standing. High school grade point averages and standardized test scores remain highly predictive of a student’s college grade point average across the wide range of Carnegie institutional classifications (Thompson, 1998).

2.4.1. Standardized Aptitude Testing Historically, high school grades and scores from standardized tests such as the Scholastic Aptitude Test (SAT) and American College Testing Program (ACT) have been used as admissions criteria for acceptance into institutions of higher education. The standardized examinations were developed to replace more subjective admissions criteria with a relatively inexpensive, standardized methodology (Hossler & Anderson, 2005; Hubin, 1997; Kubota & Connell, 1992; Pascarella & Terenzini, 2005; Rigol, 1994). According to Burton and Ramist (2002), the Scholastic Aptitude Test (SAT) and high school grades accurately predict undergraduate students’ academic performance. The researchers also confirmed a similar study by Wilson (1983) that contends that SAT scores made a substantial contribution to predicting undergraduate college students’ cumulative grade point averages. Anastasi (1988) evaluated 2000 studies using SAT scores to predict college grade point averages and surmised that SAT scores predicted 18% of the variance in freshman year students’ grade point averages. When combining SAT scores and high school grade point averages, 25% of the variance in college grade point averages is explained (Wolfe & Johnson, 1995). Despite the numerous studies demonstrating the correlation between Scholastic Aptitude Test scores and college grade point averages, several continuing research studies seem to confirm the persistence of discriminatory bias in SAT content. To address the current and past criticism, the multiple choice format currently used by the Scholastic Aptitude Test has evolved through periodic subject matter updates. Recent

19 SAT changes also have included a renewed emphasis on reasoning skills and re-centered the scoring key to better assess the current target population (Hubin, 1997). Rosser (1989) and Leonard and Jiang (1999) have suggested the Scholastic Aptitude Test under-predicts female achievement. Moffatt’s (1993) study revealed that the Scholastic Aptitude Test was not a valid predictor of freshman grades for African American students. Even though the body of literature citing SAT bias is large, higher education continues to support continued use of the Scholastic Aptitude Test as a primary tool for screening admissions applicants (Hubin, 1997; Kubota & Connell, 1992; Rigol, 1994).

2.4.2. High School Preparation A student’s high school preparation remains an important predictor of student success in higher education (House, 1995; Napoli & Wortman, 1998; Tharp, 1998; Webb, 1989). High school preparation is frequently measured by high school grades and class rank. Lenning (1982) found that the high school grade point average and class rank have higher correlations with student retention than any other predictors. Further, Wolfe and Johnson (1995) found that the high school grade point average accounted for 19% of the variance in college grade point average. Additionally, Nora (1990) and Tharp (1998) observed similar results in their studies. Finally, Pascarella, Smart and Ethington (1986) and Nora (1990) found high school grade point averages correlated with freshman student retention. The high school grade point average continues to receive significant attention as a predictive variable. Keller, Crouse, and Trusheim (1994) investigated the predictive validity of high school grades when considered separately or jointly with standardized testing. The researchers suggest that the Scholastic Aptitude Test scores add little to the predictive power of high school grades alone. Aldeman (1999) raised questions on the validity of using high school rank and the high school grade point average because the measures did not consistently measure the academic intensity of the high school curriculum. While Aldeman suggests that high school academic intensity and quality be an integral part of the admissions process,

20 higher education institutions are reluctant to invest the resources necessary to implement such a system. Contrary to Aldeman, DesJardins, Ahlburg, and McCall (2002) suggest that college preparatory coursework is an important factor in post-secondary persistence, but high school grade point average is equally effective. Research by Leonard and Jiang (1999) suggest the high school curricular bias is negligible when using high school grade point average as the sole predictor of academic achievement. However, Leonard and Jiang (1999) recommend that high school grade point average data be coupled with achievement tests to produce a more accurate achievement prediction while minimizing curricular bias.

2.5. Predictive College Variables While high school grade point averages and standardized test scores have received significant attention as predictive for college admissions, Tinto (1983) argues that these variables only measure the students’ aptitude and fail to measure the key elements for student success and retention - student engagement and motivation. Lang (2001) suggests researchers must examine additional predictors within the college environment including college grade point average, faculty-student interactions, and overall student engagement. The following section will examine each of these college predictors.

2.5.1. College Grade Point Average College grade point average is the most common measure of student success and is directly linked to continuing enrollment. In addition to being a measure of student academic performance, the college grade point average frequently serves as a surrogate measure for the degree to which students have responded favorably to the institutional environment and overall college experience (Allen, 1999; McGrath & Braunstein, 1997).

21 Tinto (1993) cites two reasons why poor academic performance frequently results in dismissal from the college environment (Tinto, 1993). First, most institutions have established policies that prevent the student whose grade point average is less than an established level from re-enrolling. Second, the low grade point average can cause a student to leave because of the negative social stigma attached to failure. While perceptions of academic integration are frequently cited as key predictors of retention, Tharp (1998), Cabrera (1993) and others found that college grade point average significantly predicted persistence rates of students (Napoli & Wortman, 1998, Nora, 1990; Webb, 1989). The college grade point average during the first year and, in some cases, first semester was found as a better indicator of continued enrollment and academic success than many other variables (Allen, 1999; McGrath & Braunstein, 1997). Additional predictors included minority status, gender, and socio-economic status. Grade point average is the primary indicator of whether students have responded to various factors that create the complex system called the college experience (Kuh, 1999; Pascarella & Terenzini, 1991). Pascarella and Terenzini (1991) asserted that a student’s grades were probably the single most revealing indicator of the student’s successful adjustment to the intellectual demands of a particular college’s course of study. Pascarella and Terenzini suggest that undergraduate grades were perhaps the single best predictor of student persistence and completion of the bachelor’s degree.

2.5.2. Role of Faculty in Retention A sense of community emerges from frequent faculty and student interactions (Fox, 1986; Moss & Young, 1995; Nora et al., 1990; Pascarella & Chapman, 1983). Students who perceived that they are engaged with faculty members are more likely to be academically successful. Being engaged with faculty and staff allows students to ask questions and establish a personal connection with others on campus (Jackson, Smith, & Hill, 2003). Students who perceive that faculty are concerned with their development stay in college (Grosset, 1991; 1993). Grosset’s research indicates that students’ perceptions of the quality of their interactions with faculty inside and outside the classroom were strong

22 predictors of persistence. Other researchers have found that frequent perceived student interaction with faculty and college personnel is significantly related to student success (Clark, Walker, & Keith, 2002; Eimers, 2001; Mayo, Murguia, & Padilla, 1995). Students need to develop meaningful relationships with faculty. The perception of supportive faculty members – those willing to help students and meet with students on a regular basis – encourages persistence (Lundquist, Spaulding, & Landrum, 2002). Faculty members that utilize class assignments and course work that encourages positive interaction with the faculty and student peers also facilitate cognitive development (Bauer & Liang, 2003; Kuh & Gonyea, 2003, Kuh & Hu, 2001).

2.5.3. Support from Other Campus Activities Similar results are seen when students seek academic assistance during office hours with other campus academic personnel such as teaching assistants (Clark, Walker, & Keith, 2002; Eimers, 2001; Grosset, 1991). In addition, students who have positive perceptions of the support they receive on campus are more likely to succeed (Brooks & DuBois, 1995). Folger, Carter and Chase (2004) found that students who participate in small group support sessions have significantly higher grade point averages than those who do not participate. However, Eaton and Bean (1995) found attending help sessions and the use of tutors did not influence academic persistence, but other behaviors including asking questions in class, meeting with faculty members, turning in assignments, and keeping track of grade performance were actions that affected students’ sense of academic integration.

2.5.4. Student Effort Several studies have examined students’ perceptions of their academic effort as it relates to academic integration and retention (Fox, 1986, Pascarella & Chapman, 1983; Nora et al., 1990). Student behaviors used as predictors of academic integration include

23 student study hours, number of books read, and participation in tutorial programs (Pascarella & Chapman, 1983), student tutoring hours, contacts with academic personnel for course assistance (Fox, 1986), frequency of library visits, and hours spent studying (Nora et al, 1990). Academic effort as a separate predictor, as measured by students’ perceptions, predicted grade point average (Strauss & Volkwein, 2002). Umoh and Spaulding (1994) found academic performance and persistence are affected by regular class attendance, class participation, and assignment completion. The amount of effort students put into their work and the time spent on class assignments reflect the degree to which they are engaged in educationally purposeful activities (Bauer & Liang, 2003). Effort on assignments contributes to greater gains over a wide range of learning and personal development outcomes. These outcomes increase the chance students will experience academic success (Kuh, 1999; Kuh & Hu, 2001) including successful grade point averages (Bauer & Liang, 2003).

The amount of time

spent on course work and turning in assignments in on time are indicators of student effort (Bauer & Liang, 2003; Kuh, 1999; Kuh & Hu, 2001). Student effort also indicates how students are adjusting to the academic demands within the institutional environment. Developing effective study skills, managing time effectively, and developing an understanding of course expectations are indicators of a student’s adjustment to the academic environment (Nisbett, Ruble, & Schurr, 1982). Students that turn in late assignments demonstrate disengagement from the institutional environment and frequently place their academic success in jeopardy (HERI, 2004).

2.6. Other Factors Influencing Academic Success and Retention Several other measures have been commonly used to determine academic success and retention including student gender and race/ethnicity. In persistence studies, gender was found to have some influence on retention efforts. Some researchers found that women were significantly more likely to persist than men (Pascarella, Duby, & Iverson, 1983; Ottinger, 1991). But others argued that the probability of dropout was not different for male and female students (DesJardins, Ahlburg, & McCall, 1999).

24 With regard to the association of race/ethnicity and probability of persisting, African Americans were more likely to drop out of college than White students (Pascarella, Duby, Miller, & Rasher, 1981). Hispanics had the same probability of dropping out of college as Whites (DesJardins, Ahlburg, & McCall, 1999). Academic achievement in high school and college was regarded as one of the key factors that helped explain the lower level or persistence of minority students, i.e. even after controlling for other factors, student race/ethnicity was an important factor in student persistence (Tinto, 1982).

2.7. Use and Growth of Course Management Systems Information technology has become pervasive within the academic and administrative activities of higher education. Specifically, the integration of eLearning tools and systems within higher education has provided a new set of opportunities for faculty members and students. eLearning is defined as the delivery of an educational program by electronic tools. eLearning tools frequently cover a wide set of technologies including web-based learning, course management systems, digital learning objects, and virtual collaboration. In only a few short years, course management systems have become an essential feature of instructional technology at institutions of higher education (Warger, 2003). Course management systems allow faculty members to develop and deliver course content, assess student knowledge, and provide communication outside of class time through a series of automated or semi-automated tools. The instructional tools within the course management systems vary within higher education, but frequently include tools for: • • • • •

Assessing student knowledge through online examinations Communicating with students through online discussions, email, and chat rooms Linking to outside resources including web sites and library materials Posting of course content (lecture notes, slides, pictures, syllabus) Tracking of student performance through an electronic grade book (Parisotto, 2004; Plattsburgh State University, 2004)

25 In October 2002, The EDUCAUSE Center for Applied Research (ECAR) surveyed 274 institutions that use eLearning tools (Arabasz, Pirani, & Fawcett, 2003). All of the respondents indicated that they had integrated some technology into face-toface, or residential instruction. Eighty-one of the respondents indicated that they had implemented courses that required the use of technology outside the classroom. Over eighty percent of the survey participants responded that they offered hybrid courses which utilized technology to reduce in-classroom time. Finally, the survey found that over seventy percent of the institutions provided at least some distance learning courses with no on-campus time. Clearly the course management system has expanded the course beyond the boundaries of the typical classroom. More recently, the 2005 Campus Computing Project survey confirms the increasingly important role of the course management system. Overall, the percentage of higher education institutions using a course management system has risen from 14.7% in 2000 to 45.9% in 2005. The increased deployment of course management systems was consistent across all sectors of post-secondary education including public and private universities, public and private four-year institutions, and two-year institutions. Less than ten percent of the respondents indicated that they provided no course management product for faculty and students. Gartner Research Group indicates the importance of course management systems remain within the top ten technologies for higher education – behind only networking and security (Yanosky, Harris, & Zastrocky, 2004). In the report of a survey conducted by WebCT (2004), 37% of the 416 respondents said they had implemented eLearning institution-wide, up from 25% in 2002. The survey also indicated that participation in eLearning was up 31% and faculty members were struggling to meet demand. The survey concluded that eLearning was part of the core of higher education and was no longer a peripheral part of higher education. According to the 2005 Campus Computing Project survey, Blackboard and WebCT remain the main course management system providers within higher education with more than 75% of the market share. As of 2000, Howell reported that WebCT had more than 6 million student accounts in 147,000 courses at more than 1,350 colleges and

26 universities in 55 countries. Michigan Virtual University (2003) reported that Blackboard was used in more than 2,700 colleges and universities in 140 countries.

2.7.1. Faculty Adoption of Course Management Systems A number of factors influence faculty adoption of course management systems. The University of Wisconsin System commissioned a study designed to determine: •

the factors which promote adoption of the course management system

•

the factors which influence faculty members to utilize additional tools within the course management system

•

the additional instructional tools needed within the course management system (Morgan, 2003)

Within this study, Morgan surveyed 730 faculty members, conducted 140 interviews, and examined course management system log files. The researcher indicated that pedagogical issues were principally underlying the faculty adoption of course management systems. Faculty would adopt the course management system based on the perceived teaching need such as providing additional practice or increasing the amount of discussion for a given topic. Carvenale (2003) cites the faculty desire to use course management systems as a tool for organizing classroom activities and content. Morgan (2003) observes that the time-saving function of a course management system has driven some faculty to adopt the course management system. Interviewees within the study stressed the efficiency and time management challenges which the course management system may solve. Peer recommendations and student requests are also a powerful factor in initiating the use of the course management system. Morgan (2003) indicated that more than twenty percent of the respondents indicated peer recommendations as the principal reason for using the course management system. From the interviews, Morgan (2003) also notes that many faculty members attributed their initial course management system adoption to their dean or other administrator. Some of the administrative push was

27 credited to the requirement for a secure electronic grade book system. While student pressure was mentioned, faculty and administrators were more likely to influence their peers in using the course management system (Morgan, 2003). Warger (2003) cites the growing amount of digital content as a driving force behind course management system usage. With an increasing amount of digital information available, faculty members are searching for effective methods to store, deliver, and monitor the usage of the materials.

2.7.2. Tool usage within course management systems Once a faculty member begins to develop a course using the course management system, a wide variety of tools ranging from discussion boards to content downloads can be selected and implemented. According to the results of a study conducted at Pennsylvania State University (Harwood, 2004), the tools selected were directly related to the reasons faculty adopted the technology. The most common reasons for adopting technology within a class included the ability to: •

provide additional content to explain topics (45.3%)

•

update lecture materials more rapidly (40.3%)

•

communicate with students outside of class (37.3%)

•

cover more course material (25.4%)

•

interact more in class (24.3%)

•

facilitate group work in and outside of class (21.7%)

Harwood (2004) summarizes by acknowledging that the course management tools were employed mainly in the delivery of course content and communication with students. A similar study at the University of Missouri-St. Louis (2001) surveyed approximately 140 instructors and found the content and communications tools were most likely to be implemented. Faculty tended to use the content tools within the course management system to supplement lecture materials and provide options for different learning styles (Carnevale,

28 2003; Parisotto, 2004). Warger (2003) reported that content tools were the highest rated features within the CMS. Fifty percent of those surveyed rated the announcement, syllabus, and content tools as very important. More than 80% of the faculty surveyed indicated they used the content tool. Further Morgan indicates that nearly 60% of faculty members believed the course management system contributed to greater contact between students and increased feedback to students on their course performance. Morgan (2003) found that faculty members are much slower to adopt the more complex or interactive tools within the course management system. However, once the tools were adopted, faculty members used the more complex tools extensively. Carnevale (2003) and Warger (2003) suggest that faculty members are reluctant to use some CMS tools because they perceive that some students are techno-phobic. For universities to help increase both academic success and retention rates, continual assessment of teaching methods as well as student demography and behavior is necessary. The students walking America’s higher education campuses today are undeniably different than their predecessors. The students beginning to inundate America’s campuses are often called the “Millennial Generation” (Carlson, 2005). These Millennial students were born roughly between the years of 1980 and 1994 and grew up in the electronic age of multi-tasking. The Millennial student does not wish to be taught in a lecture format and will rarely give full attention to any single task. Using traditional methods to educate these students does not make logical sense, hence the deployment of Course Management Systems (CMS) to help integrate technology into the higher education classroom and meet the needs of the changing student base. Higher education professors are increasing their use of CMS. In fact, based on responses from 890 colleges, more than 90% of all responding campuses reported using CMS (Carlson, 2005; Hawkins, Rudy & Nicolich, 2005). The authors of the aforementioned survey further support this claim by indicating that more money is being spent on information technology more than in previous years. In addition, more support is available for faculty using technology in teaching.

29 Colleges and universities have access to an enormous amount of data concerning their applicants and have developed sophisticated predictive admission formulas that accurately forecast student success. However, most institutions have not taken advantage of the vast volume of available data to contribute to the ongoing analysis and prediction of student success beyond the first year. More importantly, the data that CMS creates is currently not used as the valuable academic success and retention assessment tools it has the potential to be.

2.8. Summary Drawing from the significant literature regarding the purpose of higher education, this integrative literature review provides a context for the study of utilizing the course management system to predict academic success. The institution’s ethical obligation to support, educate, and provide guidance to each student it admits is at the root of the drive for increased retention (Bean, 1986). Two major college student retention theorists were reviewed in order to provide a foundation for this study. Astin (1993) and Tinto (1993) are considered the leading contributors to the body of literature regarding the college student dropout phenomenon. The literature provides a solid basis for examining academic success as it relates to retention and provides groundwork for intervention strategies that may be launched based on this study. The above research review discussed factors that influence adjustment to college and persistence. Factors that help students adjust to college while on campus include perceived faculty concern for student academic growth and success along with the availability of formal and informal opportunities for academic assistance. These factors help students meet the ongoing academic demands of their various curricula. Exposure to faculty and staff (Clark, Walker, & Keith, 2002; Eimers, 2001; Mayo, Murguia, & Padilla, 1995) is significantly related to academic success. Daily experiences within the institutional environment greatly influence academic success (Folger, Carter, & Chase, 2004).

30 College grade point averages provide a measure of whether students have been able to adjust to compounding factors that create the college experience. Low grade point averages indicate students have not responded favorably to the institutional environment or put forth the effort necessary to be successful (Allen, 1999; McGrath & Braunstein, 1997; Pascarella & Terenzini, 1991). In a review of previous course management system implementation literature, it is apparent that course management systems have become an essential feature at institutions of higher education. The review of the literature indicated that much of the use of the course management tools has focused on content presentation tools. Many of the studies cited in this chapter have found that faculty members were much slower to adopt the more advanced tools. However, once adopted, the advanced tools were heavily utilized. According to some research findings, the reasons that most faculty members start using the course management system were the pedagogical and time-saving functions. Faculty members generally found the course management system to be a good organizational tool for teaching and distributing content. A review of the literature has revealed no studies examining the use of the course management system or similar technology to examine actual (versus perceived) student behaviors. The lack of research is not surprising since the evolution of the course management systems only has recently allowed for such data mining. The purpose of this study is to expand traditional retention studies by integrating additional student behavioral data from the course management system. This proposal seeks to build upon the existing literature and identify the strongest indicators of student success within courses utilizing CMS. The author will describe how these indicators may be used to create an early-intervention scheme based on previously conducted research. It is anticipated that the utilization of course management data to predict ongoing academic success constitutes a promising step towards increasing the sophistication of current retention models and theories.

31

CHAPTER 3. RESEARCH METHODOLOGY

3.1. Research Design The study employed a causal-comparative research design. Within this study, the researcher examined the relationship between the identified independent variables and academic success. In addition, the researcher developed a model to determine if one can predict academic success in the future based on course management data. Causalcomparative is a powerful non-experimental method used when the researcher cannot manipulate the independent variables within the study or the random assignment of participants is not possible (Gall, Gall, & Borg, 2003). This means, among other things, that the variables must be observed as they occur naturally.

3.2. Participants Data for this study consisted of a convenient sample of students enrolled during the fall 2005 semester in courses utilizing the institutional course management system. The sample consisted of 71,794 course management tracking records representing 27,276 unique students enrolled in 608 courses representing 75 departments and 9 colleges at a large, research institution in the Midwest. A complete list of courses, departments, and schools is found in Appendices A.1 through A.5. The 27,276 unique undergraduate students in this sample represented 88.3% of the student population. A total of 11,071 females (40.6%) and 16,205 (59.4%) men comprised the sample which is comparable to the institutional demographics of 40.9% women and 59.1% men (Purdue University, 2005). The students’ mean age was 20.75 years as compared with the University mean of 20.6 years. Student ethnicity represented in this sample was: Caucasian, 80.4%; Asian American, 5.2%; African American, 3.5%; Hispanic American,

32 2.7%; and American Indian, 0.4%. Tables 3.1 and Table 3.2 provide a summary of the student demographics.

Table 3.1: Student demographics ____________________________________________________________________________ Demographic Minimum Maximum Mean SD Univ. Avg. ACT Composite 9 36 24.82 3.90 25.00 SAT Composite 500 1600 1130.57 148.75 1150 High School Rank (%) 1 99 74.44 18.73 78.00 Cumulative college GPA 0 4 2.76 .74 2.83 Semester college GPA 0 4 2.81 .90 2.83 Age 16.93 75.29 20.75 3.90 20.6 ____________________________________________________________________________

Table 3.2: Categorical student demographics ____________________________________________________________________________ Race Percentage University Average African American 3.5% 3.5% American Indian 0.4% 4.6% Asian American 5.2% 5.3% Caucasian 80.4% 88.1% Hispanic American 2.7% 2.7% Other 1.3% Not Reported 6.5% Gender Female 42.2% 40.9% Male 57.8% 59.1% ____________________________________________________________________________

3.3. Development of Data Subgroups Some statistical analysis assumes that the responses of different cases are independent of each other. That is, each response comes from a different, unrelated case. However in this study, multiple cases may come from a single student. For example, a student may have taken three courses during the fall semester of 2005 using the course

33 management system. As a result, the student will have three records represented within the data set with identical cognitive and demographic data. To address independence of error assumptions, two subgroups were randomly created for this study. The first subgroup was utilized in the development of the model. The second subgroup was utilized to verify the predictive model. The model development sample consisted of 27,276 course management tracking records representing 27,276 unique students enrolled in 597 courses representing 75 departments and 9 colleges at a large, research institution in the Midwest. The 27,276 unique undergraduate students in this sample represented 88.3% of the student population. A total of 11,071 females (40.6%) and 16,205 men comprised the model development sub-sample which is comparable to the institutional demographics of 40.9% women and 59.1% men (Purdue University, 2005). The students’ mean age was 20.98 years as compared with the University mean of 20.6 years. Student ethnicity represented in this subgroup was nearly identical to the larger sample and the university population. Tables 3.3 and Table 3.4 provide a summary of the student demographics.

Table 3.3: Model development subgroup - Student demographics ____________________________________________________________________________ Demographic Minimum Maximum Mean SD Univ. Avg. ACT Composite 9 36 24.96 3.94 25 SAT Composite 500 1600 1136.01 150.46 1150 High School Rank (%) 1 99 74.72 18.79 78 Cumulative college GPA 0 4 2.84 .72 2.83 Semester college GPA 0 4 2.78 .91 2.83 Age 16.93 75.29 20.98 2.59 20.6 ____________________________________________________________________________

34

Table 3.4: Model Development subgroup - Categorical student demographics ____________________________________________________________________________ Race Percentage University Average African American 3.6% 3.5% American Indian 0.4% 4.6% Asian American 5.3% 5.3% Caucasian 80.2% 88.1% Hispanic American 2.7% 2.7% Other 1.3% Not Reported 6.4% Gender Female 40.6% 40.9% Male 59.4% 59.1% ____________________________________________________________________________

The model verification sample consisted of 21,870 course management tracking records representing 21,870 unique students enrolled in 587 courses representing 75 departments and 9 colleges at a large, research institution in the Midwest. The 21,870 unique undergraduate students in this sample represented 70.8% of the student population. Student gender and ethnicity represented in the model verification subgroup was nearly identical to the larger sample and the university population. Tables 3.5 and Table 3.6 provide a summary of the student demographics.

35

Table 3.5: Model Verification Subgroup - Student demographics ____________________________________________________________________________ Demographic Minimum Maximum Mean SD Univ. Avg. ACT Composite 9 36 24.86 3.91 25 SAT Composite 500 1600 1132.57 148.82 1150 High School Rank (%) 1 99 74.50 18.78 78 Cumulative college GPA 0 4 2.81 .74 2.83 Semester college GPA 0 4 2.75 .91 2.83 Age 16.93 66.48 20.72 2.23 20.6 ____________________________________________________________________________

Table 3.6: Model Verification Subgroup - Categorical student demographics ____________________________________________________________________________ Race Percentage University Average African American 3.6% 3.5% American Indian 0.4% 4.6% Asian American 5.3% 5.3% Caucasian 80.3% 88.1% Hispanic American 2.7% 2.7% Other 1.3% Not Reported 6.4% Gender Female 41.1% 40.9% Male 58.9% 59.1% ____________________________________________________________________________

3.4. Protection of Human Subjects This study was approved by the University’s Human Subjects Institutional Review Board. This study was a secondary analysis of a data set and involved the electronic retrieval of data from two large data sets (course management and enrollment management). At no time was identifying student information made available to the researcher.

36 3.5. Independent Variables The course management system (CMS) routinely collects individual student tracking information from most tools within the system. This study utilized the twenty (20) tracking elements from the course management system as independent variables. The twenty variables were collected automatically by the system and were reported through the instructor’s “student tracking” report or harvested across numerous courses within the institution through a database query. The independent variables collected in the course management system are found in Table 3.7.

Table 3.7: Course management system independent variables

Variable

Description

Type

Assessment – time spent

The total time a student spends within the assessment tool from initiation of the tool until the student leaves the assessment or “times out” from the system. If the student if allowed to enter an assessment multiple times, the total time spent is recorded.

continuous, interval (hh:mm:ss)

Assessments completed

The number of assessments completed by the student.

discrete, interval

Assessments opened

The total number of assessments opened by the student. If a student opens the same assessment multiple times, the system records each entry.

discrete, interval

(table continues)

37

Variable

Description

Type

Assignments – time spent

The total time a student spends within the assignment tool from initiation of the tool until the student leaves the assignment or “times out” from the system. If the student if allowed to enter an assignment multiple times, the total time spent is recorded.

continuous, interval (hh:mm:ss)

Assignments completed

The number of assignments completed by the student.

discrete, interval

Assignments opened

The total number of assignments opened by the student. If a student opens the same assignment multiple times, the system records each entry.

discrete, interval

Calendar entries created

The total number of calendar entries created by the student.

discrete, interval

Calendar entries read

The total number of calendar entries opened by the student. If a student opens the same calendar entry multiple times, the system records each entry.

discrete, interval

Chat and whiteboard sessions

The number of chat and whiteboard sessions in which a student participated within the course.

discrete, interval

Content files viewed

The total number of content files opened by the student. If a student opens the same content file multiple times, the system records each entry.

discrete, interval

Discussion postings created

The total number of discussion postings created by the student within the course.

discrete, interval

(table continues)

38

Variable

Description

Type

Discussion postings read

The total number of discussion postings opened by the student. If a student opens the same discussion posting multiple times, the system records each entry.

discrete, interval

Email messages read

The total number of email messages opened by the student. If a student opens the same email message multiple times, the system records each entry.

discrete, interval

Email messages sent

The total number of email messages sent by the student within the course.

discrete, interval

Media library collections viewed

The total number of media library collections opened by the student. If a student opens the same media library collection multiple times, the system records each entry.

discrete, interval

Media library entries viewed

The total number of media library entries opened by the student. If a student opens the same media library entry multiple times, the system records each entry.

discrete, interval

Organizer pages viewed

The total number of organizer pages opened by the student. If a student opens the same organizer page multiple times, the system records each entry.

discrete, interval

Sessions

The total number of times the student enters the course. No minimum length of time spent in the course is required.

discrete, interval

(table continues)

39

Variable

Description

Type

Total time

The total amount of time the student spends logged into the course. Time is recorded from entry into the course until the student leaves the course or is “timed out” by the system.

continuous, interval (hh:mm:ss)

URL’s viewed

The total number of URL’s opened by the student. If a student opens the same URL multiple times, the system records each entry.

discrete, interval

In addition to the course management variables, the study utilized fifteen (15) variables from the enrollment management system. The fifteen variables were collected through the admissions process. The independent variables collected in the enrollment management system are found in Table 3.8.

Table 3.8: Demographic and cognitive independent variables Variable

Description

Type

Subject

The host department from which the course is offered.

discrete, nominal

Course

The course number of the course.

discrete, nominal

Course Grade

The final course grade of the student. Entries are A, B, C, D, F, I, or W. If the student drops the course within the official drop/add window, the course grade field will be null.

discrete, ordinal

High School Rank

The high school rank as expressed as a percentile.

Continuous, ordinal (table continues)

40

Variable

Description

Type

SAT Verbal Score

The numeric SAT verbal score.

Continuous, ordinal

SAT Math Score

The numeric SAT mathematics score.

Continuous, ordinal

ACT Composite Score

The ACT composite score.

Continuous, ordinal

Birth Date

The birth date of the student (month, day, year).

Continuous, interval

Race

The race of the student (selfreported).

discrete, nominal

Gender

The gender of the student (selfreported).

discrete, nominal

Full-time or Part-time Status

Code for full-time or part-time student based on the number of credit hours currently enrolled.

discrete, nominal

Class Code

The current academic standing of the student as expressed by the number of semesters of completed coursework. Ranges from one to eight for undergraduate students. One (1) indicates a first semester freshman. Four would indicate a second semester sophomore.

discrete, ordinal

Cumulative Grade Point Average

Cumulative university grade point average (four point scale).

Continuous, ordinal

Semester Grade Point Average

Semester university grade point average (four point scale).

Continuous, ordinal

University Standing

Current university standing such as probation, dean’s list, or semester honors.

discrete, nominal

41 3.6. Dependent Variables As stated earlier, the purpose of this study was to develop a model utilizing a combination of course management tracking data and demographic variables that can accurately predict academic success within a course. Academic success was measured by the final course grade and was the dependent variable in this study. Course grades were selected since college grades have the most influence on student persistence (Porter, 1990). The intent to persist is highly related to college grade point average (Cabrera, Nora, & Castañeda 1993). In this study, academic success was examined through three related approaches. The first approach was studied through the final course grade - A, B, C, D or F. The second approach defined academic success as those getting a grade of “C” or higher. The final approach defined academic success as those getting a grade of “B” or higher. The three approaches will provide campuses with alternative methods for implementing an intervention system.

3.7. Preparation of Data for Analysis The study’s data was collected ex-post facto from records collected during the fall semester, 2005. Student participation in the study was based on the faculty usage of the course management system. The data from the course management system was exported by the information technology staff and delivered to the university’s identity management office. The university’s identity management office converted the course management accounts into a university identifier (equivalent to a social security number). Identity management delivered the file to the university’s enrollment management office. Demographic and cognitive data was integrated with the course management system data. Finally, all identifying information was removed from the data file and the file delivered to the researcher. The entire process took approximately eight weeks.

42 3.7.1. Removal of Extraneous Records Extraneous data was removed from the overall data set. Extraneous data was defined as records without corresponding final course grades and records with students having a graduate classification as defined by the class code. Two institutional processes allowed for the creation of information within the course management system, without corresponding course grade information. First, the institution allowed the faculty member to create “guest” accounts within their courses in the system. While guests have full privileges within the course, and subsequently are tracked within the system, they have no official status within the institution and thus do not receive an official course grade. Second, based on institutional policy, students are not removed from the course management system even if the student drops the course. The policy prevents student records including grades from being deleted within the system for momentary withdrawal from the course either by the student’s action (formal withdrawal from the course) or by the institution (failure to pay tuition). Thus, students that drop a course within the normal drop-add period are still listed in the course management system, but do not have a course grade for the given course. All records without a corresponding grade (including withdrawals) were removed from consideration in this study. Students not officially classified as an undergraduate student within the institution were removed from consideration in this study. The removal included all records without a class code from one to eight. Students outside of this range include graduate students and non-degree seeking students. The original data set from the course management system consisted of 85,414 records and was narrowed to 71,794 records by utilizing the criteria above.

3.7.2. Treatment of Missing Data A review of incomplete information was conducted in accord with procedures for missing data (Tabachnick & Fidell, 2001). Close attention was paid to determine whether incomplete information was missing systematically or completely at random. This was done to ascertain whether to use a complete case or replacement approach.

43 Data omitted systematically is missing at random (MAR) and reflects a bias in the absent value of Y that appears to have been influenced by X. The observed values of Y have a higher level of randomness in data missing completely at random (MCAR). The complete case approach is the remedy available for data to be MAR (Tabachnick & Fidell, 2001). Statistical Package for the Social Sciences (SPSS) provides a technique designed specifically for MAR data – imputation via maximum likelihood estimation. Maximum likelihood estimation calculates an estimated valued based on the sample data. SPSS also provides several complete case solutions for MCAR data including “list-wise” to delete missing cases and “pair-wise” to exclude variables for the omitted values. SPSS can also create dummy values for MCAR data. By default, SPSS logistic regression does a list-wise deletion of missing data. This means that if there is a missing value for any variable in the model, the entire case will be excluded from the analysis. The research data contained a large percentage of missing data within the course management independent variables. Based on normal educational practice, faculty members select the number of tools utilized within their course. While some faculty member may use a majority of the tools within the course management system, other faculty members may only utilize one or two tools. Due to the large number of “null” or missing values, the researcher has eliminated the use of course management tool data from inclusion in this study. Table 3.9 provides a summary of missing cases by independent variable.

44

Table 3.9: Summary of missing cases Variable

Course Management Variables Assessment – time spent Assessments completed Assessments opened Assignments – time spent Assignments completed Assignments opened Calendar entries created Calendar entries read Chat and whiteboard sessions Content files viewed Discussion postings created Discussion postings read Email messages read Email messages sent Media library viewed collections Media library viewed entries Organizer pages viewed

Entire Sample (n=71,794)

Model Development Subgroup (n=27,276)

82.29% (59,078) 84.97% (61,005) 84.97% (61,005) 82.72% (59,386) 82.50% (59,228) 78.97% (56,696) 68.65% (49,288) 9.60% (6,894) 100.00% (71,794) 28.93% (20,770) 72.54% (52,076) 71.43% (51,285) 41.51% (29,802) 100.00% (71,794) 100.00% (71,794) 100.00% (71,794) 0.30% (218)

82.81% (22,586) 85.02% (23,191) 85.02% (23,191) 84.44% (22,586) 84.20% (22,967) 80.50% (21,958) 69.90% (19,066) 10.32% (2,816) 100.00% (27,276) 29.59% (8,071) 73.27% (19,984) 71.89% (19,984) 43.27% (11,803) 100.00% (27,276) 100.00% (27,276) 100.00% (27,276) 0.31% (85)

Model Verification Subgroup (n=21,870) 82.93% (18,136) 85.58% (18,716) 85.58% (18,716) 83.02% (18,156) 82.21% (18,110) 79.19% (17,318) 68.97% (15,083) 9.77% (2,136) 100.00% (21,870) 29.80% (6,518) 73.11% (15,990) 72.02% (15,750) 42.30% (9,250) 100.00% (21,870) 100.00% (21,870) 100.00% (21,870) 0.33% (72) (table continues)

45

Variable

Sessions Total time URL’s viewed Enrollment Management Variables Subject Course Course Grade High School Rank SAT Verbal Score SAT Math Score ACT Composite Score Standardized Test Birth Date Race Gender Full-time or Part-time Status Class Code Cumulative Grade Point Average Semester Grade Point Average University Standing

Entire Sample (n=71,794) 0.00% (0) 1.35% (968) 57.90% (41,567)

Model Development Subgroup (n=27,276) 0.00% (0) 0.00% (0) 59.30% (11,803)

Model Verification Subgroup (n=21,870) 0.00% (0) 0.00% (0) 58.16% (12,720)

0.00% (0) 0.00% (0) 0.00% (0) 20.03% (14,284) 17.86% (12,819) 17.79% (12,775) 61.82% (44,386) 4.23% (3,036) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (1) 0.00% (0)

0.00% (0) 0.00% (0) 0.00% (0) 19.92% (5,434) 18.18% (4,960) 18.13% (4,949) 61.74% (16,840) 4.63% (1,262) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0)

0.00% (0) 0.00% (0) 0.00% (0) 20.04% (4,382) 17.73% (3,878) 17.67% (3,864) 61.62% (13,476) 4.09% (894) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0) 0.00% (0)

46 3.7.3. Calculated Variables Based on the original data set, several variables were calculated or developed to facilitate additional analyses. The constructed variables can be found in Table 3.10.

Table 3.10: Calculated independent variables Variable Academic success

Age Aptitude score

Composite SAT score Course completion

CMS tool usage

Description For the purposes of this study, academic success was defined as students completing the course within the normal timeframe and receiving a grade of C or better. The birth date was converted to an age of the student as of January 1, 2006 as expressed in years. The aptitude score was defined as the SAT composite score or the converted ACT to SAT score. In the cases in which students have both SAT and ACT scores, the SAT score will remain. The composite SAT score was defined as the sum of the SAT verbal and SAT math scores. Course completion was defined as students completing the course within the normal semester timeframe. In other words, students who did not withdrawal or receive an incomplete. The course tool usage was defined as the number of tools a faculty member utilizes within the course. For a course tool to be counted, at least fifty percent of the students must have utilized the tool at least once. The range of tool usage was from one to ten.

Type discrete, interval

continuous, interval discrete, interval

discrete, interval discrete, nominal

discrete, interval

47 For the purposes of this study, student success was defined as the following: completing the course within the normal timeframe of the semester and receiving a grade of C or above. Course completion was defined as a student that completes the course within the normal timeframe and receives a grade of F or above. Students receiving a W (withdrawal) or I (incomplete) were not considered to have completed the course. In addition to the calculated variables in Table 3.10, all time-based variables were converted from hours, minutes, and seconds to the number of hours calculated to one decimal place.

3.7.4. Coding of Independent Variables for Analysis In Table 3.11, a summary of the coding utilized for the analysis of all the variables is identified.

Table 3.11 Coding of independent variables Variable Academic success

Coding 0 = Grade of D or F 1 = Grade of C or better

Class Code

1 = First Semester Freshman 2 = Second Semester Freshman 3 = First Semester Sophomore 4 = Second Semester Sophomore 5 = First Semester Junior 6 = Second Semester Junior 7 = First Semester Senior 8 = Second Semester Senior

Course completion

0 = Course not completed within semester 1 = Course completed within semester (table continues)

48

Variable Course Grade

Coding 4=A 3=B 2=C 1=D 0=F null = I or W

Course Size

0 = Small (250)

Gender

1 = Female 2 = Male

Race

1 = African American 2 = Asian American 3 = Caucasian 4 = Hispanic 5 = Native American 6 = Other 7 = Not Reported

Full-time or Part-time Status University Standing

1 = Full-time student 2 = Part-time student 0 = Probation 1 = Regular standing 2 = Semester honors 3 = Semester honors and deans list

3.7.5. Standardization of Variables Due to the nature of the research, the researcher cannot control, nor desires to control, the use of the course management system by the students or the instructors. All course management data elements were transformed into Z scores. The resulting transformed scores will have a mean of zero and a standard deviation of one.

49

Ζ=

X −x SD

Formula 3.1 Calculating a Z score

The Z score for an item indicates how far and in what direction that item deviates from its class distribution's mean. Z scores are expressed in units of the distribution's standard deviation. X is the value of the independent variable, µ is the class mean for the independent variable, and σ is the class standard deviation for the independent variable.

3.7.6. Removal of Outliers An outlier is a data point distinct from the rest of the data. Outliers may occur due to input or measurement errors, or may be due to normal variations of a sample. Procedures for detection of outliers rely on the detection of extreme residuals and through visual examination of the data (Pallant, 2005). Outlying cases were identified through the Statistical Package for the Social Sciences (SPSS) and evaluated for removal prior to data analysis.

3.8. Research question 1: Correlational analysis The study first examined if the independent variables supplied by WebCT Vista course management system were correlated with student success in undergraduate courses at a large, research university. Bivariate correlational techniques were used to analyze the degree of the relationship between two variables. All independent variables were compared with the dependent variable to determine the strength of the correlation. In addition, correlations between all independent variables to verify statistical assumptions were made prior to conducting factor analysis and multinomial logistic regression. The form of the variables to be correlated determined the technique which

50 was used. A summary of correlational techniques used in this study is shown in Table 3.12 (Gall, Gall, & Borg, 2003).

Table 3.12: Bivariate correlational technique Type

Categorical

Continuous

Dichotomous

Rank

Categorical Continuous Dichotomous Rank

Polychoric Polyserial Polychoric Polyserial

Pearson Polyserial Polyserial

Polychoric Polyserial

Polyserial

3.9. Research question 2: Reduction of Variables Due to the large number of independent variables, the researcher examined which independent variables in the WebCT Vista course management system would be combined to predict student success in undergraduate courses at a large, research university. Two techniques were utilized to reduce the overall number of independent variables – reduction through affiliation and reduction through factor analysis.

3.9.1. Reduction through Affiliation A number of the course management independent variables are different measures of the same course management tool. As a result, different measures of the same course management tool were combined into a single measure. For example, assessment time spent, assessments completed, and assessments opened were combined into an “assessment composite” value by calculating the mean of the three original values. The composite variables can be found in Table 3.13.

51

Table 3.13: Course management system composite variables Variable

Components

Calculation

Assessment composite

Assessment – time spent Assessments completed Assessments opened

Mean value of standardized usage

Assignments composite

Assignments – time spent Assignments completed Assignments opened

Mean value of standardized usage

Calendar composite

Calendar entries created Calendar entries read

Mean value of standardized usage

Discussions composite

Discussion postings created Discussion postings read

Mean value of standardized usage

Email composite

Email messages read Email messages sent

Mean value of standardized usage

Media library composite

Media library collections viewed Media library entries viewed

Mean value of standardized usage

3.9.2. Reduction through Factor Analysis In an attempt to reduce the large number of independent variables within this study, exploratory factor analysis (EFA) was utilized to uncover the underlying interrelationships of a relatively large set of variables (Pallant, 2005). Factor analysis is a statistical technique applied “to a single set of variables when the researcher is interested in discovering which variables in the set form coherent subsets that are relatively independent of one another (Tabachnick & Fidell, 2001, p. 582). Prior to performing the factor analysis, the suitability of data was assessed through the inspection of the correlation matrix to insure the presence of a number of coefficients greater than 0.3. The Kaiser-Meyer-Oklin value was examined to determine

52 if the variables exceed the recommended value of 0.6 (Kaiser, 1970, 1974). The Barlett’s Test of Spericity (Bartlett, 1954) was examined to determine statistical significance.

3.9.2.1. Assumptions of Factor Analysis Most statistical tests rely upon certain assumptions about the variables used in the analysis. Therefore, the following section focuses on the assumptions for factor analysis as identified by Tabachnick and Fidell (2001) including sample size, linearity, absence of multicollinearity, and the factorability of R. 3.9.2.1.1. Sample Size Correlations are less reliable when estimated from small sample sizes. Tabachnick and Fidell (2001) provide a guide for sample sizes with a general rule of having at least 300 cases for factor analysis. With the data set containing 71,794 cases, and the sub-samples containing over 20,000, sample size was not an issue in this study. 3.9.2.1.2. Linearity Factor analysis assumes a linear relationship among variables. Tabachnick and Fidell (2001) suggest that linearity be examined through scatter plots, however the large number of cases makes this approach impractical. Correlational analysis was run between independent variables, and those correlations below 0.3 were spot checked through scatter plots. 3.9.2.1.3. Absence of Multicollinearity Multicollinearity is an issue derived from having a correlation matrix with too high of a correlation between independent variables. In factor analysis, Tabachnick and Fidell (2001) suggest examining all independent variables with eigenvalues approaching 0.

53 3.9.2.1.4. Factorability of R Factor analysis depends on correlations between independent variables. Correlations were computed between all independent variable, and values less than 0.30 were reconsidered (Tabachnick & Fidell, 2001).

3.9.2.2. Extraction of the Factors After examining the presence of outliers, absence of multicollinearity, and factorability of the correlations matrix, nineteen independent course management variables were subjected to principal components analysis (PCA) using Statistical Package for the Social Sciences (SPSS) version 12. Factors were extracted using a direct oblimin (oblique rotation) method. The results were further supported through a Parallel Analysis to identify components with eigenvalues exceeding the corresponding criterion values for a randomly generated data matrix of the same size - 19 variables x 71,794 records (O’Connor, 2000).

3.10. Research Question 3: Predicting Student Success Multinomial logistic regression was used to predict which data elements in the WebCT Vista course management system and enrollment management system combine to predict student success in undergraduate courses at a large, research university. Logistic regression was selected because the researcher needed to: •

predict course grade on the basis of continuous and/or categorical independent variables

•

determine the percent of variance in the course grade explained by the independent variables

•

rank the relative importance of independent variables

•

assess interaction effects of the independent variables

Logistic regression applies a natural log transformation (ln) to the equation in Figure 3.2. The dependent variable (academic success) becomes the log of the odds that

54 a particular choice will be made (Kachigan, 1991). The logistic regression model is described mathematically as follows: ln (P/1-P) = ln (odds) = α + β1X1 + β2X2 + β3X3 + β4X4 + β5X5… + ε Formula 3.2: Logistic regression equation P is the probability that a student will receive a particular grade and 1-P is the probability that the student did not receive a particular grade. X1, X2, X3, etc. are the independent variables within this study. Alpha (α) and beta (β) are the regression coefficients to be estimated, and epsilon (ε) is a random error term that is logistically distributed (Kachigan, 1991).

3.10.1. Assumptions of Logistic Regression Most statistical tests rely upon certain assumptions about the variables used in the analysis. When these assumptions are not met, the results may not be trustworthy, resulting in a Type I or Type II error, or over- or underestimation of significance or effect size(s). As Pedhauzer (1997, p. 33) notes, "Knowledge and understanding of the situations when violations of assumptions lead to serious biases, and when they are of little consequence, are essential to meaningful data analysis." Therefore, the following section will focus on the assumptions for logistic regression identified by Tabachnick and Fidell (2001) including the ratio of cases to variables, adequacy of expected frequencies, absence of multicollinearity, absence of outliers, and independence of errors.

3.10.2. Ratio of Cases to Variables Problems may occur when there are too few cases relative to the number of predictor variables. Logistic regression will produce extremely large parameter estimates and standard errors when too many cells have no cases (Tabachnick & Fidell, 2001). With the large data set, the large number of continuous independent variables may result

55 in a large number of cells with no cases. To reduce the number of empty cells, several strategies were utilized as previously identified in section 3.6.2.

3.10.3. Adequacy of Expected Frequencies Logistic regression may have too little power if expected frequencies are too small. Tabachnick and Fidell (2001) suggest that all expected frequencies are greater than one and no more than 20% of the cells are less than five. Again, the researcher utilized the strategies previously identified in section 3.6.2.

3.10.4. Absence of Multicollinearity Multicollinearity (frequently simplified as collinearity) are issues which are derived from having a correlation matrix with too high of a correlation between independent variables. Multicollinearity exposes the redundancy of variables and the need to remove variables from the analysis (Grimm & Yarnold, 2005). Several methods were utilized to identify multicollinearity – correlations between independent variables, high variance inflation factor (VIF), and the tolerance statistic. For the purpose of this study, independent variables with correlations greater than 0.8 or a variance inflation factor (VIF) greater than 4.0 were removed from consideration in the model (Grimm & Yarnold, 2005; Pedhauzer, 1997).

3.10.5. Independence of Errors Logistic regression assumes that the responses of different cases are independent of each other. That is, each response comes from a different, unrelated case. To eliminate multiple records from a given individual, sub-samples were created through random selection.

56 3.10.6. Significance Tests for Logistic Regression Prior to conducting the multinomial logistic regression, the data set was divided randomly into two subsets. The first subset was utilized to create the overall model. The second subset was used to determine the overall predictive reliability of the model (research question 4). An evaluation to group means for each of the independent variables was conducted to ensure no significant difference between groups. Numerous indices are available for logistic regression to determine the significance of the model. This study will examine the log likelihood ratio, Chi-square test of goodness-of-fit, and effect size.

3.10.6.1. Likelihood Ratio The likelihood ratio test is based on the log likelihood which is specifically the probability that the observed values of the dependent may be predicted from the observed values of the independents. The likelihood ratio can be used for assessing the significance of logistic regression and is analogous to the use of the sum of squared errors in ordinary least squares (OLS) regression. The likelihood ratio was used to test the overall model and test individual parameters. SPSS was used to report the likelihood ratio for each predictor as well as for the overall model.

3.10.6.2. Chi-square Test of Goodness-of-Fit The Chi-square test of goodness-of-fit determines if the sample under analysis follows the same distribution as the population from which the sample was selected. The test evaluates the null hypotheses H0 (that the sample follows the population distribution) against the alternative (that the data are not drawn from the population). If chi-square goodness-of-fit is not significant, then the model has adequate fit (Tabachnick & Fidell, 2001). By the same token, if the test is significant, the model does not adequately fit the

57 data. The study used the test as it appears in SPSS multinomial logistic regression output in the "Goodness-of-Fit".

3.10.6.3. Effect Size The effect size, R2, is utilized as a measure of the efficiency of the regression by providing a measure of the amount of variance accounted by the equation (Kachigan, 1991; Tabachnick & Fidell, 2001). A number of measures have been proposed to approximate R2 measures for multinomial logistic regression. These are approximations since the R2 measure seeks to make a statement about the "percent of variance explained." However, all logistic psuedo-R2 measures attempt to measure strength of association and not goodness-of-fit and are reported as approximations to R2, not as the actual percent of variance explained (Tabachnick & Fidell, 2001). This study provided three measures of psudeo-R2 (Cox & Snell, Nagelkerke, & McFadden) as reported by SPSS. Each of these measures provides an indication of the amount of variation in the dependent variable explained by the model (Pallant, 2005).

3.11. Research Question 4: Predictive Power of the Model The final research question of this study was to determine the predictive power of the proposed model to predict student success in undergraduate courses at a large, research university. Multinomial logistic regression was implemented according to the same procedures in 3.9 but with the second data subset created in section 3.9.6. Comparison of group results were reported including the overall predictive accuracy of the model, the amount of Type I (misidentifying as failing when they are not) and the amount of Type II error (misidentifying as successful when they are not).

58 3.12. Reliability Reliability is the consistency of measurement within the study. More simply, reliability is the degree to which an instrument provides repeatable results each time it is used under the same conditions with the same subjects. For this study, the reliability for the course management, cognitive and demographic data, and the course grade will be discussed. All course management data was collected by a common central server. Due to the nature of the system, all data collected by the course management system is highly reliable since it is collected automatically and can not be changed by the researcher nor the system users. The demographic and cognitive data may vary in reliability. Some independent variables such as SAT and ACT scores are highly reliable and have extensive statistical measures in place to insure reliability (Ewing, Huff, Andrews & King, 2005). Other independent variables such as high school grade and rank may have issues with internal consistency and inter-rater (school) reliability. However, it should be noted that all high school information was reported directly from the high school through transcripts and was not influenced by the individual student. For this reason, SAT or ACT scores were preferable for use as an independent variable. Finally, the final course grade could be called into question. The degree of interrater reliability could be a significant issue. Despite the limitations and flaws of utilizing grades as a measure and predictor of academic success, previous research has found that grades remain the single best predictor and largest contributor to student persistence, degree completion, and graduate school enrollment (Pascarella & Terenzini, 2005, DesJardins, Ahlburg, & McCall, 1999).

3.13. Validity Validity is the strength of the study’s conclusions. More formally, Kachigan (1991) defines validity as “the extent to which our measurements reflect what we intend them to, or what we claim they do” (p. 140). External validity involves the degree to which the study is measuring what it claims to measure.

59 For this study, content validation involves understanding the independent variables by examining the underlying processes used to collect the data. As a first step, a data dictionary was created for the entire data set. A course management specialist, who had been working on the system, was contacted to confirm the interpretation of the study variables described within the data dictionary. The data dictionary assisted the researcher in making proper interpretations of course management data. For example, one can not assume “time” actually measures quality time. The course management system only measures time a user is “logged into” the system, but not necessarily engaged within the content. Another example includes the number of discussion postings “read,” the data can include a student reading the same posting multiple times. In order to improve validity, the researcher cross validated the results. With a large number of records, the researcher had the flexibility to divide the data set into two subsets without impacting the overall power of the study. As a result, one sub-data set was utilized in the model development, while the other subset was utilized for model testing and validation. External validity examines if the study’s results can be generalized to other populations, settings or time periods. The determination of external validity is outside the scope of this study. Since the study is retrospective in nature, utilizing data that was previously collected, the tools available to the researcher took several steps to reduce the threats to validity. The study assumed that the course management system data was captured correctly and consistently. The researcher used subsets of data to test the overall validity of the model and developed a data dictionary to reduce overall misinterpretations of the data.

3.14. Hypothesis Testing – Type I and Type II Error The end goal of the study was to develop a model to predict academic success based on data from the course management system and enrollment management. As with any hypothesis testing, there are two types of error. The first type of error is classifying academically successful students as failing (Type I error). The second type of error is

60 classifying the academically failing students as successful (Type II error). Type I error is defined as the probability of rejecting the null hypothesis when it is true – in other words, Type I error is finding things that are not there. Type II error is the probability of failing to reject the null hypothesis when it is false – in other words, Type II error is failing to find things that are there (Tabachnick & Fidell, 2001). For the purposes of this study, the researcher selected a model that minimizes the Type II error because the impact of misclassifying students as successful when they are in trouble is viewed as the greater risk. Both prediction success tables and classification tables will be used to determine Type I and Type II error.

3.15. Limitations of Study The purpose of this study, and of causal-comparative research, is to provide evidence that one or more independent variables predict the change in the dependent (Gall, Gall, & Borg, 2003). As a result, the key problem in non-experimental research is that an observed relationship between an independent variable and a dependent variable may not be a causal relationship; rather it is a relationship that is the result of the operation of a third variable (see Figure 3.1). For an example, academic performance (dependent variable) may be related to the number of sessions. However, that relationship may be largely due to the joint influence of the third variable of computer availability.

Independent Variable (computer ownership)

Independent Variable (sessions)

?

Dependent Variable (academic success)

Figure 3.1: The issue of cause and effect

61 The study investigated data obtained at only one research university in the Midwest during the fall semester of 2005. The defined size of the institution and region of the country may contain samples of students that are skewed due to individual population characteristics. Students in higher education are not a homogenous population, nor are the institutions which serve them. Institutions vary greatly in size and mission, and there is significant variation among academic disciplines. As a result, making generalizations to other higher education institutions may not be possible. The study employed a secondary analysis of a large data set. Thus the research will be limited by pre-collected variables. Since the collection of the data set was done prior to the start of the study, the researcher had no control over the collection methodology or definition of the independent variables. The study only examined students enrolled within courses in which the faculty members elected to use the course management system. As a result, the study may neglect students that would not fit well within the developed model. However, with such a large percentage of the undergraduate population (88.3%), number of courses (608), and variety of departments (75) represented, it was unlikely that any major population was missed in this analysis. Finally, another limitation is the course management system. The institutional course management system used in this study is currently used by more than 2000 institutions throughout the world. However, other course management systems may collect different tracking information. As a result, the comparison with samples from other course management systems may not be valid.

3.16. Summary Secondary data analysis using a causal comparative design was used to predict the academic success, as defined by course grade, of undergraduate students in courses utilizing the course management system. Unique to this study was the inclusion of course management data (behavior data) with traditional cognitive and demographic data. All data for this study was retrieved from two large data bases – the course management system and the enrollment management data warehouse.

62 The purpose of this chapter was to describe the methodology to be used to conduct the study and answer the research questions examining the prediction of academic success using data from the course management system and/or enrollment management. Statistical analyses will include factorial analysis and multinomial logistic regression, as well as descriptive analyses, including frequencies, percentages, means, and standard deviations.

63

CHAPTER 4. RESEARCH RESULTS

The results presented in this chapter are organized into several major sections. The first section is an examination of the characteristics of academic success within this study. The second section describes the results of the first research question which explores the correlation of course management variables with academic success. The third section focuses on the reduction of the twenty course management variables into five factors. The fourth section presents the analysis of the third research question utilizing regression analysis to determine the ability of the variables to predict academic success. The final section of this chapter explains the predictive validity of the model (research question four).

4.1. Characteristics of Dependent Variables As stated earlier, the purpose of this study was to develop a model utilizing a combination of course management tracking data and demographic variables that can accurately predict academic success within a course. Academic success was measured by the final course grade and was the dependent variable in this study. Nearly two-thirds (62.64%) of the students within this sample earned a course grade of an A or B, while only 15.23% were considered academically unsuccessful by earning a failing grade of a D or F. The complete distribution of grades is provided in Table 4.1.

64

Table 4.1: Number of students receiving each course grade Grade A B C D F Total

Numeric Grade 4 3 2 1 0 Total

Number of Students 8,424 8,664 6,034 2,089 2,065 27,276

Percent 30.88% 31.76% 22.12% 7.66% 7.57% 100.00%

Minority students comprised nearly twenty percent of the study sample (19.7%). Asian American and African American students comprised over half of the minority population. The association of race/ethnicity and probability of persisting has been studied by a number of researchers. Pascarella et. al. (1981) found African Americans were more likely to drop out of college than Caucasian American students. In this study, the analysis found a significant, but small effect between minority and non-minority students (χ2= 209.76, df=4, p