A path analysis of first-year social science students’ engagement with their degree and Level 1 academic outcome

Dr Carl Walker, Stephanie Fleischer and Dr Sandra Winn, Department of Applied Social Science, University of Brighton

Abstract

The expansion of higher education in the UK and attempts to widen participation have changed the context of undergraduate learning. This study examines student engagement with their degree. Quantitative data for 388 UK Level 1 students were used to develop a path analysis model of the relationship between Level 1 academic performance, gender, academic engagement, attendance and prior university entry points. Structural equation modelling allowed a detailed understanding of the direct and indirect effects of key variables that contribute to Level 1 learning outcomes, and findings are discussed within the context of structural changes to Level 1 student engagement imperatives with a view to improving the learning experience for all students.

Introduction

The expansion of higher education in the UK and attempts to widen participation have changed the context of undergraduate learning and teaching. Rapid growth in student numbers has greatly increased class sizes in many institutions (Universities UK 2001), and the student body has become more diverse, in terms of both demography and educational experiences prior to entry (Department for Education and Skills [DfES] 2003). Expansion of the system has, in part, been funded by shifting some of the costs of higher education to students and their families, and one consequence of this has been a substantial growth in student employment during term-time. For some students, term-time employment now occupies more time than academic work (Centre for Higher Education Research and Information and London South Bank University 2005).

At the same time as these policy developments, the learning and teaching environment has been altered by the growing use of new learning technologies. Virtual learning environments (VLEs) are now widespread in higher education institutions (Joint Information Systems Committee 2004) and are increasingly employed by academic staff to provide learning materials and interactive forms of learning such as electronic discussions (Kirkwood and Price 2005).

Student performance and attendance

Several studies have demonstrated a significant positive correlation between attendance and degree result (e.g., Romer 1993; Lamdin 1996; Woodfield et al. 2006). Borland and Howsen (1998) suggest that attendance, rather than being important in its own right, may be a proxy for other factors such as ability. However, research investigating the predictive effects on academic performance of both attendance and measures of ability have shown that attendance has an independent effect on academic outcome (Romer 1993, Gatherer and Manning 1998, Rogers 2001, Woodfield et al. 2006) although the extent of the relative importance of attendance has been debated (Devadoss and Foltz 1996, Gatherer and Manning 1998, Hunter and Tetley 1999). That said, there is generally little research on the causes of student absenteeism (Longhurst 1999) and Woodfield et al. (2006) suggested a general need for further research on attendance in higher education. Prior grade average is also believed to contribute to academic performance, with performance attained prior to college a reliable predictor of grade point average during a semester (Devadoss and Foltz 1996, Plant et al. 2005).

Other factors influencing student engagement

Studies conducted in the USA have usually found no significant association between number of hours engaged in study and academic performance (Schuman et al. 1985; Plant et al. 2005). Simply counting the number of study hours does not capture the complexity of students’ study patterns.
Hoskins and van Hooff (2005) analysed 110 undergraduates in the second year of a psychology degree and found that bulletin-board use influenced academic achievement, with those posting messages outperforming those not using or passively using bulletin boards. However, again, further work is needed to understand the relationship between student engagement in VLEs and academic performance.

Gender is another factor that has been shown to influence academic outcome. Gender has been shown to be related to degree achievement, (Simonite 2003), with women more likely to get good degrees than men although this may not be a trend followed for all ages (Richardson and Woodley 2003). The more focused learner identity of females is believed to lead them to work harder and more consistently. Generally, however, little research in the field has focused on student attendance rates and gender differences (Woodfield et al. 2006).

The rationale

Most of the research in the UK concerning student engagement and independent study has investigated the student experience of study, rather than attempting to quantify its relationship with academic outcomes. This work has revealed the difficulties that many students experience in adapting to a mode of learning and teaching which is at odds with the more didactic concepts of learning developed from earlier stages of their educational careers (Kember 2001).

In this paper, we disentangle the effects of some of the prominent variables which are thought to contribute to academic outcome. Path analysis and structural equation modelling allow researchers to look at the interrelation between direct and indirect effects on outcome variables and so has obvious utility for this work. We explore first-year students’ engagement with their academic work. Our aim is to quantitatively examine the relationship between measures of engagement with academic work, prior educational attainment, gender and Level 1 academic outcome.

Methodology

Participants

The sample comprised a group of students who completed Level 1 of the undergraduate programme in applied social science at the University of Brighton (n = 388). This was all of the students on our social science courses. These students study degrees in criminology, sociology, social policy, health and social care, social science and psychology, usually combining two subjects for a joint honours degree. Table 1 provides demographic and academic information on the sample. Our sample was strongly weighted toward females, but this is a good representation of the gender divide of students on our social science courses and many social science courses in the UK. Moreover, we have sufficient numbers of both men and women to include gender as a variable in the model.

Table 1: Demographic and academic constitution of Level 1 sample

Procedure

Data about prior educational attainment, lecture attendance, engagement with information technology (IT) learning materials, gender and Level 1 academic outcome were obtained. Data on academic performance was an average of students’ scores on the six modules that they undertook that year. Attendance data was collected from a compulsory module, ‘Social Science Research Methods’, which is taken by all first-year students. Prior educational attainment was a measure based upon the number of points scored through the UK Universities and Colleges Admissions Service (UCAS) accreditation system (UCAS 2006). Entry points were obtained from students’ records.

Students’ engagement with independent study can be more difficult to measure. With the advent of VLEs in higher education, it has become possible to obtain empirical measures of some elements of students’ independent study activity. Many VLEs enable academic staff to track students’ usage of learning materials made available to students electronically. In the present study, engagement with IT learning materials was measured using the University of Brighton ‘StudentCentral’ system, which provides information and learning materials for students throughout the course. This particular measure was taken from the module on ‘Social Science Research Methods’ in Semester 2, which is taken by all Level 1 students on the applied social science degrees. This course was used since it had the most comprehensive development and use of IT learning materials.

Students who withdrew during the year were excluded from the study because their student records were incomplete.

Design and analysis

Preliminary analysis of this data was conducted to examine bivariate associations among the academic performance and academic engagement variables.

Exploratory path analysis was then used to examine the relationships between Level 1 academic outcome and gender, engagement and previous academic achievement. Path analysis allows the simultaneous estimation of multiple regression equations and provides estimates of direct and indirect impact of these variables on stipulated other variables in a model. Essentially, this means that we will be able to tease out in detail the matrix of relationships between the variables. Furthermore, path analysis allows the separation of direct and indirect associations between the endogenous and exogenous variables. Exogenous variables in a path model are those with no explicit causes whereas endogenous variables include intervening causal variables and dependent variables. So, as well as understanding the way in which Variable A directly influences Level 1 academic outcome, we can also look at the way in which Variable A influences Level 1 academic outcome through the way that it would influence a third variable, Variable C, which also influences Level 1 academic outcome. It is essentially an extension of the simple regression model where one dependent variable is modelled by a number of independent variables. The model allows for intervening endogenous variables that can in turn affect other endogenous variables (Neeleman et al. 2004).

Model fitting was hypothesis-driven, and exogenous variables were chosen following close scrutiny of the literature on student engagement and academic outcome (Devadoss and Foltz 1996; Hunter and Tetley 1999; Rogers 2001, Simonite 2003, Hoskins and van Hooff 2005, Woodfield et al. 2006). As such, it was exploratory rather than confirmatory.

The hypothesized model was run using Amos 6.0. As suggested by Bollen and Long (1993), several goodness-of-fit measures were used to test the fit of the model. This means that we have a number of ways of understanding how well the model of our predictor variables influenced Level 1 academic outcome. Chi-square was generated, although with large samples even trivial differences between data and model can give large chi-square values and unwarranted model rejection. The Comparative Fit Index (CFI) is derived from the comparison of a hypothesised model with the independence model (the model in which variables are assumed to be uncorrelated with the dependent variable) and has been suggested as an index of choice (Byrne 2000). The Tucker-Lewis Index (TLI) is consistent with such measures as the CFI and also compares the specified model with that of the independence model. For both the TLI and the CFI, values of above 0.95 are considered to indicate superior fit (Byrne 2000). Finally, the root mean square errors of approximation (RMSEA) provides sample-size adjusted estimates indicating good fit when smaller than 0.05. Path coefficients allow us to understand the extent to which predictor variables influenced Level 1 academic outcome and were fully standardised and therefore comparable within the model.

Sample size

It is recommended that the number of participants to the number of parameters is estimated at the ratio of 10 to 1 (Kline 2005). Our hypothesised model had 23 parameters with a sample size of 388. This yielded a ratio of 17 to 1, which is thus acceptable.

Results

End-of-year academic performance was significantly correlated with gender, measures of academic engagement and UCAS points (Table 2). There were also significant associations between all other pairs of variables except UCAS points and IT engagement. It can be seen from Table 2 that the highest association (r = .439) is between Level 1 academic outcome and prior educational attainment.

Table 2: Pearson product-moment correlations for the study variables

The model in Figure 1 shows the direct and indirect effects of the independent variables on Level 1 academic performance. The model resulted in a good fit without modification indices (Table 3 shows that TLI and CFI were above 0.95, Chi-square was not significant, and RMSEA was less than 0.05 [Byrne 2000]) and so can effectively account for Level 1 academic performance. Based on the findings of our model, the following paths were non-significant: gender to Level 1 outcome directly and gender to attendance. These were annotated with a dashed line. Since the model is exploratory and theory-dependent, non-significant effects have not been eliminated (Abd-El-Fattah 2006).

Figure 1: A structural equation model of standardised path coefficients for student attendance, IT engagement, prior educational attainment, gender on level 1 academic achievement. Dashed lines indicate non-significant path coefficients

Table 3: Goodness of fit indices of models

Direct effects are when one variable directly relates to a second variable whereas indirect effects are where one variable relates to another variable through its influence on a third variable. Figure 1 highlights both the key direct and indirect effects of the independent variables. Table 4 shows the total effects, direct effects and indirect effects of the independent variables on Level 1 academic outcome with 95 per cent bias-corrected confidence intervals as generated by the AMOS 6.0 programme through the bootstrapping technique (Arbuckle 2003). This process allows us to effectively estimate indirect effects and their 95 per cent confidence interval. This table allows us an estimation of standardised indirect and direct effects of the independent variables on Level 1 academic outcome.

Table 4: The standardised direct and indirect effects of antecedent variables on Level 1 academic outcome

PC = Path coefficient, CR = Critical ratio for each path coefficient, this is obtained by dividing the estimate by its standard error (using a significance level of 0.05, any critical ratio that exceeds 1.96 in magnitude is said to be significant), LB = Low boundary of bias-corrected confidence interval of bootstrapping, HB = High boundary of bias-corrected confidence interval of bootstrapping. Values shown in italic font indicate non-significant direct effects.

Discussion and implications

The difficulties that some students have with engaging in their academic work are often exacerbated by the new study skills required in higher education, such as note-taking and academic reading (Sutherland et al. 2002, Drew 2001). The model was significant and so the factors involved require further consideration. The model supported previous research (Devadoss and Foltz 1996; Woodfield et al. 2006) in showing that engagement, as measured by attendance at lectures, was an important factor in predicting Level 1 academic performance. Not only was it important through directly influencing exam score but also through indirectly through the influence that attendance has on the commitment to VLEs. Such environments are a growing feature in the undergraduate pedagogical structure, and issues that address attendance are paramount. Since we know that simply recording attendance increases both attendance and academic performance (Shimoff and Catania 2001), new approaches to understanding the reasons that our students do not attend, and to facilitate attendance, are essential.

Prior academic grades influenced Level 1 outcome both directly and indirectly through attendance. This was not unexpected (Plant et al. 2005, Devadoss and Foltz 1996), but the suggestion that prior educational attainment had little impact on the relationship between attendance and performance (Romer 1993) was not borne out. It was important to include previous academic performance in the model such that we could obtain the relationship between the engagement variables independent of previous achievement.

A number of authors have confirmed the finding that gender is related to degree achievement (Simonite 2003, Richardson and Woodley 2003) and that female students perform better in exams generally than male students (Gatherer and Manning 1998). Our model shows that gender influenced academic outcome via previous academic performance. It would appear that gender is associated directly with academic performance to a greater extent when younger than when at university Level 1.

Limitations

While this work has thrown interesting insight upon student engagement and the interrelation of some of the key factors that influence student performance, it should be considered in the context of some of the key limitations. This research only used students from whom we were able to derive a measure of prior academic success as defined by the UCAS tariff. A minority of students, often mature students, arrive through different courses that do not provide an easy measure of previous academic success. Further work with this population would be a useful addition to this work. Moreover, there may have been differences between the students who withdrew from their studies and those who continued, and, so, future work might also investigate this.

Engagement with virtual learning materials was taken from a Level 1 course on research methods, which all students took. Whilst it provided the best source of data and utilisation of the StudentCentral facility, it may be that engagement patterns could have been specific to this course. Moreover, lecture attendance and the extent to which students engaged with virtual learning materials were specific indices of student engagement and, as mentioned earlier, simply counting the number of study hours does not capture the complexity of students’ study patterns. As such, perhaps future research should include multiple indices of student engagement in order to enrich the data. It would also useful to apply this model with a less ethnocentric and female sample.

However, overall this work has provided important intuitions and an innovative working model that has separated some of the key factors that combine to influence Level 1 academic outcome. It has also provided a fertile context for further, more detailed investigation.

References

Abd-El-Fattah, M. (2006) ‘Effects of Family Background and Parental Involvement on Egyptian Adolescent’s Academic Achievement and School Disengagement: A Structural Equation Modelling Analysis’, Social Psychology of Education, 9 (2): 139–57.

Arbuckle, J. (2003) AMOS 5.0 Update to the Amos User’s Guide, Chicago, Ill.: SPSS.

Bollen, K. A. and Long, J. S. (1993) ‘Introduction’, in K. A. Bollen and J. S. Long (eds), Testing Structural Equation Models, Newbury Park, Calif.: Sage Publications, pp. 1–9.

Borland, M. and Howsen, R. (1998) ‘Effect of Student Attendance on Performance: Comment on Lamdin’, Journal of Educational Research, 91 (4): 195–7.

Byrne, B. M. (2000) Structural Equation Modelling with AMOS: Basic Concepts, Applications and Programming, Englewood Cliff, NJ: Lawrence Erlbaum Associates.

Centre for Higher Education Research and Information and London South Bank University (2005) Survey of Higher Education Students’ Attitudes to Debt and Term-Time Working and Their Impact on Attainment, London: Higher Education Funding Council for England and Universities UK.

Devadoss, S. and Foltz, J. (1996) ‘Evaluation of Factor Influencing Student Class Attendance and Performance’, American Journal of Agricultural Economics, 78 (3): 499–508.

Drew, S. (2001) ‘Student Perceptions of What Helps Them Learn and Develop in Higher Education’, Teaching in Higher Education, 6 (3): 309–31.

Department for Education and Skills (2003) The Future of Higher Education, Cm 5735, London: The Stationery Office.

Gatherer, D. and Manning, F. C. R. (1998) ‘Correlation of Examination Performance with Lecture Attendance: A Comparative Study of First-Year Biological Sciences Undergraduates’, Biochemical Education, 26: 121–3.

Hoskins, S. L. and van Hooff, J. C. (2005) ‘Motivation and Ability: Which Students Use Online Learning and What Influence Does It Have on Their Achievement?’ British Journal of Educational Technology, 36 (2): 177–92.

Hunter, S. and Tetley, J. (1999) ‘Lectures: Why Don’t Students Attend? Why Do Students Attend?’ HERDSA Annual International Conference, Melbourne, 12–15 July.

Joint Information Systems Committee (2004) JISC Strategy 2004–6, Bristol: Joint Information Systems Committee. Available online at http://www.jisc.ac.uk/index.cfm?name=strategy_jisc_04_06. (Accessed 24 July 2008.)

Kember, D. (2001) ‘Beliefs About Knowledge and the Process of Teaching and Learning As a Factor in Adjusting to Study in Higher Education’, Studies in Higher Education, 26 (2): 205–21.

Kirkwood, A. and Price, L. (2005) ‘Learners and Learning in the Twenty-First Century: What Do We Know About Students’ Attitudes Towards and Experiences of Information and Communication Technologies That Will Help Us Design Courses?’ Studies in Higher Education, 30 (3): 257–74.

Kline, R. B. (2005) Principles and Practice of Structural Equation Modelling, 2nd edn, New York: The Guilford Press.

Lamdin, D. (1996) ‘Evidence of Students’ Attendance As an Independent Variable in Education Production Functions’, Journal of Educational Research, 89 (3): 155–62.

Longhurst, R. J. (1999) ‘Why Aren’t They Here? Student Absenteeism in a Further Education College’, Journal of Further and Higher Education, 23 (1): 61–75.

Neeleman, J., Bijl, R., and Ormel, J. (2004) ‘Neuroticism, a Central Link Between Somatic and Psychiatric Morbidity: Path Analysis of Prospective Data’, Psychological Medicine, 34: 521–31.

Plant, E., Ericsson, K., Hill, L. and Asberg, K. (2005) ‘Why Study Time Does Not Predict Grade Point Average Across College Students: Implications of Deliberate Practice for Academic Performance’, Contemporary Educational Psychology, 30 (1): 96–116.

Richardson, J. T. E. and Woodley, A. (2003) ‘Another Look at the Role of Age, Gender and Subject a Predictors of Academic Attainment in Higher Education’, Studies in Higher Education, 28 (4): 475–93.

Rogers, J. R. (2001) ‘A Panel-Data Study of the Effect of Student Attendance on University Performance’, Australian Journal of Education, 45 (3): 284–6.

Romer, D. (1993) ‘Do Students Go to Class? Should They?’ Journal of Economic Perspectives, 7 (3): 167–74.

Schuman, H., Walsh, E., Olson, C. and Etheridge, B. (1985) ‘Effort and Reward: The Assumption That College Grades Are Affected by Quality of Study’, Social Forces, 63 (4): 945–66.

Shimoff, E. and Catania, A. C. (2001) ‘Effects of Recording Attendance on Grades in Introductory Psychology’, Teaching of Psychology, 28 (3): 192–5.

Simonite, V. (2003) ‘A Longitudinal Study of Achievement in a Modular First Degree Course’, Studies in Higher Education, 28 (3): 293–302.

Sutherland, P., Badger, R. and White, G. (2002) ‘How New Students Take Notes at Lectures’, Journal of Further and Higher Education, 26 (4): 377–88.

Woodfield, R., Jessop, D. and McMillan, L. (2006) ‘Gender Differences in Undergraduate Attendance Rates’, Studies in Higher Education, 31 (1): 1–22.

UCAS (2006) The UCAS Tariff, Cheltenham: UCAS.

Universities UK (2001) Spending Review 2002: Investing for Success, Universities UK’s Submission, London: Universities UK.

The authors

Carl Walker gained his Ph.D. in Health Psychology at London Metropolitan University. His research has been focused around mental health, service user involvement and action research. As a senior lecturer at Brighton University, his research and teaching have focused around social inequality and mental distress. It has also taken on an educational component with a specific interest in social transitions in higher education.

Stephanie Fleischer is a lecturer in the School of Applied Social Science at the University of Brighton with the integrated role of a student support and guidance tutor for first-year undergraduates. A key part of her role is to help students with their transition to university, to improve retention and to understand factors affecting withdrawals. Her research interests are in student retention, first-year student experience and student finance.

Sandra Winn was a principal lecturer in social policy at the University of Brighton from 1988. She was a quantitative research methods specialist who had a specific interest in education policy and higher education. She made a major contribution to our understanding of the effect of introducing fees into higher education on students’ well-being and lifestyle. In particular, she evidenced the problems created when students have to undertake paid work in order to survive. Sandra recently passed away.