A study of factors affecting pass rates of level 5 written assessments for undergraduate adult nursing students

In the UK, undergraduate students studying for a BSc in Adult Nursing are assessed using a range of strategies, which include an examination of their ability to critically discuss and articulate their understanding of specific topics through written pieces of work such as essays. The academic level that needs to be demonstrated to achieve a pass mark of 40% increases each year of the 3-year course, from level 4, to level 5 and then finally level 6. Students who fail to achieve this usually have only one more chance to resubmit before it impacts their ability to progress.

During the second wave of the pandemic, Northumbria University adopted a blended-learning approach for first-year adult nursing students. This included increased flexibility in clinical skills teaching, student supervision and assessment. To ensure that COVID-19 restrictions did not disadvantage nursing students, the Nursing and Midwifery Council (2020) implemented emergency regulations that allowed simulated learning to be used towards the 2300 clinical placement hours required for registration. When these students progressed to the second year, for the first semester, some of these restrictions were still in force and so they continued to be taught using a blended-learning approach prior to submitting two level 5 written assessments. However, during the marking and assessment process, the academic teaching team noticed that the assessment marks for these students were significantly lower than in previous years. Additionally, the failure rate increased from 11% to 23%, even though there were no changes in module content, team or delivery. These assignments represented the students' first experience at level 5, and informal feedback from markers highlighted concerns about academic writing.

To address this, it was essential to analyse existing data to uncover patterns contributing to the elevated failure rate in level 5 academic assessments, which require a pass mark of 40%. The goal was to empower module and programme teams to enhance student support, thus optimising their chances of progressing in their nursing programme.

Background

To examine the current understanding of why academic grades can decrease when assessments remain unchanged, a review of the current literature was undertaken through a variety of databases and search engines. These included CINAHL, Web of Science and Google Scholar. Key search terms included: ‘nursing students’, ‘student nurses’, ‘baccalaureate academic performance’, ‘COVID-19’, and ‘baccalaureate academic grades’.

Key themes identified included feelings of being underprepared, stress and digital poverty.

Feelings of being underprepared and stress

Nursing students have been found to have higher levels of anxiety than traditional students (Gallego-Gómez et al, 2020). In the present study, this was combined with the additional stressors of being part of frontline care during a pandemic. There were fears of the risk of spreading the disease to family and concerns over assessments and exam readiness. It is not surprising that several studies during the pandemic sought to investigate students' academic experiences and identify factors causing further anxiety.

A global university approach included a rapid transition to online teaching via virtual learning platforms due to campus shutdowns. In a US study, 58% of participants strongly agreed or agreed that the pandemic affected their ability to successfully complete their studies (Emory et al, 2021). Comparcini et al (2022) found that more than 88% of students had experienced some level of anxiety and this was linked to concerns about grade achievement, which was higher in second-year students. Conversely, Rohde et al (2022) found that first-year students were more vulnerable and concluded that they were more anxious and worried about achieving their educational outcomes than the other years. Alomari et al (2021) highlighted concerns about students' academic progression, as 23.9% of respondents felt that more responsibility for their learning had been placed upon them, due to online learning in a home environment. This made them feel more isolated because of a lack of interaction with teaching staff and peers.

The quality of educational journeys was a major concern. A Norwegian study concluded that student satisfaction and concerns were directly linked to how universities handled the pandemic and the move to online learning (Flølo et al 2022). A Spanish study found that students overwhelmingly agreed that it was less easy to concentrate during virtual teaching sessions and self-reported poor-to-fair academic performance, which left them feeling dissatisfied (Oducado and Estoque 2021). Furthermore, Emory et al (2021) argued that student perceptions of exam unpreparedness could lie with their possible attribution to poor quality teaching because of the move to online teaching. An increased academic workload was also a theme in a study by Kells and Mathis (2023), who found that 80% of participants acknowledged additional stress due to changes in coursework. This mirrored results from Oducado and Estoque (2021), who found that more than 90% of their participants felt online learning was more stressful.

Savitsky et al (2020a) discovered that female students had higher levels of anxiety, and this could be attributed to their additional caring responsibilities for family members. However, a follow-up study found that anxiety levels decreased as restrictions eased (Savitsky et al, 2021b). Similar findings arose in Gallego-Gómez et al's study (2020), which also found that students with pre-existing emotional, financial or personal problems tended to have increased stress levels during the pandemic; however, academic performance improved with the change to online learning.

Digital poverty

Students could encounter digital poverty, in terms of wi-fi connectivity, availability of high-speed broadband (Stringfellow, 2021) and access to hardware such as laptops or desktop computers (Agu et al, 2021). Although most British households have fixed broadband access (Ofcom, 2022a), Barber (2021) highlighted that inadequacies exist in the availability and reliability of digital infrastructure (broadband and wi-fi). Seah (2020) claimed that every tenth household does not have access to the internet. Ofcom (2022b) has suggested that inequality is associated with location, as the availability of high-speed broadband and internet speed differ between rural and urban areas. The Office for Students (OFS) (2020) reported that more than 52% of students experienced a slow or unreliable internet connection and 8% were severely impacted by the lack of a connection. Poor broadband is therefore seen as an obstacle to learning because having access to the internet is considered an educational necessity (Bramley, 2021; Butcher and Curry, 2022).

Butcher and Curry (2022) highlighted the hidden costs of online learning, which Seah (2020) explained are not only the cost of a fixed broadband connection and the availability of adequate megabytes, but also those associated with obtaining and maintaining a mobile phone data plan and credit, and charging devices. Holmes and Burgess (2020) described the link between poverty and digital exclusion/poverty, meaning that during lockdowns widening learning gaps existed for the poorest or most vulnerable groups, creating further barriers to widening educational participation.

Butcher and Curry (2022) posited that educators cannot assume that all students have access to electronic devices. The OFS (2020) suggested that approximately 20% of students had problems accessing an electronic device, whether it be a laptop, tablet or personal computer. During the pandemic, access issues were exacerbated due to university and school closures and home working, as many households had to share devices, and some students were left to access online learning via mobile phone data plans (Butcher and Curry, 2022). Consequently, higher education institutions must think of long-term solutions to ensure that equity and quality of learning are maintained (Barber, 2021).

It is clear that the factors contributing to student disengagement from learning are well known, but there is a knowledge gap regarding which students are impacted and how this affects their academic performance.

Methods

Aims

This study's primary aim was to identify any correlation between antecedent variables and the marks received by students for the level 5 assessments in semester 1 of their second year. Demographic data, specifically the age and sex of students, were taken as a starting point for this. Whether each student had a disabled student support recommendation (DSSR) in place, for conditions such as dyslexia, was also recorded. By doing this, it was intended to ascertain whether any specific demographic had particularly struggled to adjust to online learning or had done so particularly well. It was noted that students began the adult nursing programme with a wide array of qualifications sufficient to meet the Universities and Colleges Admissions Service (UCAS) points requirement. It was initially intended to compare the performance of those with different entry qualifications to ascertain whether this predicted performance. However, the combinations of different qualifications proved so vast and complex that this was considered unlikely to provide any meaningful results. In additions to considering the demographic data, the study sought to identify any evidence of disengagement by students during the period of online learning and ascertain whether any correlation existed between levels of engagement and assessment results.

Finally, the study sought to ascertain whether a change occurred during semester 2. This period corresponded with a return to campus and a higher proportion of face-to-face teaching. However, given the faculty's observations regarding a generally poor standard of academic writing, it also corresponded with the introduction of an online resource package aimed at supporting academic writing.

Study design

A retrospective cohort design was used. As the need for this study became apparent following the results of the first semester modules, a retrospective research design was necessary. Retrospective studies are noted to be particularly useful in the study of rare phenomena (Song, 2010; Talari and Goyal, 2020). Regarding this study, the high failure rate had not been observed in previous cohorts, nor could it be assumed that it would occur in those that followed. It was, therefore, necessary to consider what features of this specific cohort, in the unique context of the COVID-19 pandemic, were associated with the noted outcomes.

Talari and Goyal (2020) identified two types of retrospective study. The first is the case-control study, which seeks to define cases by varying outcomes, then collects and analyses their exposure to an independent variable to establish a relationship. The second is the cohort study, which follows the progress of a group with a specific trait or exposure, and then follows that cohort over time, either prospectively or retrospectively, to measure the frequency of outcomes to identify any associations. This study took the latter approach, seeking to distinguish between students based on specific traits or exposures and ascertain whether any association with results was obtained. Song (2010) stated that, in this design, the exposure is identified retrospectively. In this study, several possible exposures would be considered as having a connection with the outcome of academic marks. However, as the study collected data on all students in an academic year, this also allowed comparison with students without that specific trait or exposure; this allowed the logic of the case-control design to be used to compare the possible influence of the exposure on marks achieved.

In a cohort study, the occurrence of an outcome can be measured prospectively or retrospectively (Talari and Goyal, 2020). In this study, the primary outcome – performance in the semester 1 assessment block – was recorded retrospectively. However, this did not preclude a prospective dimension also being included because it also attempted to identify whether a change occurred, and this was explored by collecting semester 2 results.

Demographic data and module results were already in the possession of the researchers; however, the concept of engagement was more nebulous. Three indicators of this concept that could be retrospectively collected were identified:

• Whether the student had received a warning regarding their attendance. These were issued to students who had not accessed university systems for more than 7 days
• The number of online supported learning packages (OSLs) accessed by each student for the skills-based module. These were intended as precursors to taught sessions. The completion of each was recorded on the module website
• The date of last access to the university's Skills for Practice website was recorded. This website included many skills taught in the module, students were directed to it by the module site, and it contained relevant material to the module assignment.

Therefore it was considered to be an indicator of engagement. The research team collected data from electronic systems such as SITS e:Vision, Arc-Net, Blackboard, and the Skills for Practice website. They also liaised with the student engagement team to ascertain which students had not engaged with university systems and received an attendance warning. Once collected, data were organised into a variable-by-case data grid and subjected to bivariate and non-parametric testing to ascertain any correlations with outcome in semester 1. Once this was complete, associated variables were also compared to semester 2 outcomes to ascertain whether differences in results persisted over time. All data was analysed using the statistical package IBM SPSS and predictive analytics software.

Sampling and recruitment

All second-year adult nursing students who submitted assignments were included in the study. Retrospective studies can suffer from selection bias. By including all students it was intended that this would be avoided.

Ethical considerations

Ethical approval for the study was granted by the university's Faculty Ethics Committee (reference 45360). Specific student consent to access and use data was not required for this research project as access to the data was granted by the students when they enrolled on the university nursing programme.

Results

Participant characteristics

Data from second-year adult nursing students who submitted written assignments in semester 1 of the academic year beginning September 2021 were included (n=265). Data including age, sex, online supported learning access, assessment marks, access data for the Skills for Practice website, attendance data, and the existence of a DSSR were collected.

Demographics

Analysis began with the calculation of the age of each student at the beginning of 2021, and a combined mark (out of 200) for the semester 1 modules that they had undertaken. Kendall's Tau was employed to ascertain any possible correlation between continuous data collected and semester 1 results. It was chosen because the variables such as ‘age’ did not follow a normal distribution, and this test is considered suitable when many pieces of data share identical rank. A small (0.196) but significant (P<0.001) positive correlation indicated that age was relevant to the marks received by students, but one that only partially accounted for the variance in marks. The small proportion (6%) of male students received a higher mean mark than the female students (122 compared to 107). This difference was significant (Mann-Whitney U Test P=0.033); however, this reflected the fact that the male students were also significantly older (Mann-Whitney U Test P=0.004). When divided into age bands, no significant difference between the performance of male and female students within the same bands was found.

Engagement

Further testing was therefore undertaken to explore the impact of different levels of engagement. A two-step clustering technique was used to identify different patterns of engagement with the programme. This was chosen because it allowed the inclusion of categorical data, such as whether or not the student had received a warning regarding attendance during taught sessions, along with continuous data, such as the number of online supported learning packages with which each student had engaged. The year in which the student had last engaged with the university's Skills for Practice website was also included because it contained material useful and relevant to the module assignment.

A two-cluster solution was initially produced. It immediately became apparent that the clustering process had simply divided all students on the sole basis of whether they had received an attendance warning. A three-cluster solution was therefore specified to discern differences based on other parameters. This resulted in 27.2% in cluster 1, 42.6% in cluster 2 and 30.2 % in cluster 3 (Table 1).

This solution had a 0.4 index of cohesion and separation, which SPSS describes as ‘fair’. As cluster analysis can impose a structure on data, regardless of whether this structure actually exists, it was necessary to verify that meaningful clusters had been found and to establish their meaning. Frequencies and descriptive statistics were used for this, along with non-parametric testing.

No students in cluster 1 or 3 had received an attendance warning, while all students in cluster 2 had had one. However, cluster 2 did appear more engaged than cluster 1 on other parameters: a higher percentage had accessed the Skills for Practice website during their second year and, on average, they had completed more OSLs too. However, neither cluster had engaged with the online resources to the extent of cluster 3. The Kruskal-WallisTest identified significant differences (sig=<0.001) between each of the three clusters on the parameter of the number of OSLs completed. Cluster 1, therefore, attended taught sessions but made little use of online resources. Cluster 2 had the worst attendance record for taught sessions but made greater use of online resources. Cluster 3 also attended taught sessions and made the most use of online resources. An immediate insight was that the intuitive expectation, that those who attended the fewest taught sessions would also make the least use of online resources, was wrong.

Next, it was considered whether these differing levels of engagement impacted first-semester module results. Descriptive statistics suggested this possibility, as cluster 3 (117.7) received a higher mean mark for combined semester 1 results than cluster 1 (102.4) or cluster 2 (105.0). Again, Kruskal-Wallis testing was used to compare the distribution of marks across the clusters. Although the significance (sig=<0.001) confirmed the difference in distribution, pairwise comparison demonstrated that this difference existed between cluster 3 and the others, but not between cluster 1 and 2. Finally, it was noted that cluster 3 was, on average, older (26.6 years) compared with cluster 2 (23.9 years) and cluster 1 (23.1 years). However, these differences were not statistically significant (sig=0.095). This may be due to the relatively small number of students aged 30 years or over (n=47). However, 44.7% of students aged 30 years or over were found to be in cluster 3, compared with 27% of students under this age. The chi-square test proved the difference to be significant (P=0.041).

Student disability and accessibility

A final area investigated was the performance of students with a DSSR.Twenty-five such students were identified, who were spread almost proportionally across the clusters (32.2%, 40% and 28% respectively). Students with a DSSR in place performed significantly worse in the semester 1 modules, receiving a mean combined mark of 96.44, compared with 109.37 received by those without a DSSR. This proved significant (Mann-Whitney U Test P =0.024). However, within individual clusters, a significant difference between students with DSSRs and those without were only found in cluster 2: students who had received attendance warnings (Mann-Whitney U Test P=0.017). No significant differences were found within clusters 1 and 3.

Progression over time

A package of resources on academic writing, literature searching and referencing were developed and released to students following the first semester results. For the second semester, most teaching was delivered on campus, rather than in the virtual classroom. Second-semester results were transformed into a combined mark in the same way as first-semester results. The change in each student's mark was also calculated by subtracting the semester 1 mark from the semester 2 mark. Some improvements in the marks received by clusters 1 and 2 were noticeable and, while cluster 3 continued to receive the highest average marks, a narrowing in the difference of the mean score between clusters was also perceived (Table 2).

The difference between these clusters in terms of marks received noted during semester 1 had become insignificant over semester 2 (sig=0.189). However, a more pronounced improvement was visible among students who failed one or more semester 1 modules. A new classification was calculated of the ‘progressing student’:

• Passed both semester 1 modules
• Passed one and failed one semester 1 module
• Failed both semester 2 modules (Table 3).

The difference in the distribution of change in mark across these groups was found to be significant (sig<0.001), although only between those who had passed both previous modules, and the other two clusters.

Finally, it should be noted that the previous significant difference between the performance of students with DSSRs and those without resolved during the second semester. Although the mean mark of DSSR students remained slightly lower, this difference was now insignificant. However, the improvement among DSSR students was significantly higher than among non-DSSR students (Mann-Whitney U Test P=0.048). On investigation within clusters, significant improvement among students with DSSRs compared with others was only found in cluster 2 (Mann-Whitney U Test P=0.039).

Discussion

One area within the findings that first revealed some statistical relevance was in relation to the age of the students, as older students achieved mean higher marks than younger students. Although this was a small correlation it was further reinforced by the fact that a small proportion (6%) of male students also received higher marks than female students, but this seemed to be because they were significantly older, rather than due to their biological sex. Age rather than sex was also found to be a factor for determining academic performance in a study by McCarey et al (2007), who found no statistical significance between gender and academic performance, but positive correlations between the age of students and their grades and entry-level qualifications and overall performance. Although entry-level data were collected in the present study, analysis was not undertaken due to the complexities associated with the number of variations in the qualifications found. However, this would be an avenue that would benefit from further exploration for future studies – and more simplified categorical system would need to be put into place for data retrieval.

Studies conducted over the past 20 years have shown a continuous pattern of mature students performing at a higher level than their younger peers (Houltram, 1996; Kevern et al, 1999; Ofori, 2000; Fernandez et al, 2012; Nagelsmith et al, 2012). Shanahan (2000) suggested that this is because many mature students, especially on professional courses, see this as a huge stepping stone for changing their lives and, for some, the last chance to prove themselves. Older students tend to seek additional academic support and have higher levels of self-efficacy and an internal locus of control, which can aid in their academic success (Ofori and Charleton, 2002; Glew et al, 2019). Hayden et al (2016) argued that academic success within this age group is due to both intrinsic (life experience and motivation) and extrinsic (support from peers and family) factors; however, they also highlighted that many studies examining age and academic success lack consensus on what constitutes being classified as a mature student. For many studies, mature status is selected for students over the age of 21 years, while for others it is those aged over 26 years. For this study, we examined the mean ages within the three clusters, against two age subsets, those over 30 years of age and those aged over 40 years, which clearly demonstrated that cluster 3 had a higher percentage of students within those two age ranges.

Nevertheless, age only partially accounted for the variance in marks because the findings also determined that those students with greater levels of engagement (within cluster 3) received a higher mean mark for combined semester 1 results than cluster 1 (102.4) or cluster 2 (105.0). When reviewing the students within cluster 3, what was also worthy of note was that this cluster was populated by a greater percentage of older students than the other two cluster groups and therefore there needs to be an acknowledgement that age and engagement could have contributed to the increase in marks within this specific group.

For this study, engagement was analysed from multiple perspectives, and Sattar et al (2018) also stated that, due to the changing shape of contemporary education, engagement is best defined and viewed via multiple and diverse variables. For cluster 3 students who achieved the highest marks, it was clear that they attended taught sessions and made use of the online resources; whereas students within clusters 1 and 2 did not fully engage in at least one element that was used for analysis. This finding supports previous findings from McCarey et al (2007), who stated that nonattendance at taught sessions is a significant individual predictor of poorer performance. The relationship between lecture attendance and academic marks was also the focus of a study by Doggrell (2021), who concluded that students who attended 50% or more of lectures achieved higher academic outcomes than those who attended less than 50% of lectures, and this was greater for the nursing students than for the non-nursing students.

Cluster 3 students also took more advantage of the online resources, and this is contrary to the previously held misconception that mature students have negative attitudes and are less engaged with learning that involves technology (Bramer, 2020). A recent study by Staddon (2020) also suggested that this is not the case. Findings highlight that older students are confident when using alternative and new approaches – they just use fewer technologies for more specific targeted activities than those who are younger.

There is also a well-known phenomenon of ‘second-year blues’ or ‘sophomore slump’, described as a decline in academic performance between year 1 (level 4) and year 2 (level 5) (Witherspoon, 2022). It can therefore be expected that some second-year nursing students may have increased anxiety around their academic performance during this period.

Addressing a deficit in nursing students' academic literacy was discussed by Chu et al (2012), particularly when students transition from academic level 4 to level 5. In their small pilot study using evaluation research, they delivered several sessions around different elements of academic literacy, before asking students to complete pre- and post-tests. Findings revealed that students expressed increased confidence in their abilities and there was a clear improvement in their academic grades in the subsequent semester. Considering the widening participation agenda and growth in non-traditional students, these findings are particularly pertinent.

The last finding, which is worthy of note, relates to students with a DSSR in place who performed significantly worse in the semester 1 modules, receiving a mean combined mark of 96.44, compared with 109.37 received by those without a DSSR. It is possible that due to the pandemic and the lack of contact, students with learning needs were negatively impacted by a reduction in guidance and peer support. This study revealed that the previously noted significant difference between the performance of students with DSSRs and those without resolved during the second semester. Although the mean mark of DSSR students remained slightly lower, this was now an insignificant difference. Chu et al (2012) also found that, when academics take additional steps to prepare students for academic writing and assessments, students' confidence increases, which can positively influence grades.

Limitations

There are several limitations in this study, which are worth identifying and discussing as various criticisms of retrospective research design exist. Although retrospective studies may identify associations and possible relationships between variables, Talari and Goyal (2020) and Song (2010) stated that claims regarding causation based on retrospective studies should not be made. Retrospective research designs are vulnerable to bias (Song, 2010), and Talari and Goyal (2020) specifically identified the risk of selection bias, advocating stringent selection criteria to avoid this. However, as every student in this cohort of adult nursing students was included in the present study, the authors suggest that the sample was necessarily representative.

Song (2010) identified that, in retrospective studies, data are never initially collected for the purpose of the research project; instead, the researchers are limited to data that has already been collected for other purposes. Talari and Goyal (2020) specifically noted that unrecognised confounding factors can exist when data are collected for reasons other than research. In this case, data collection was limited to data already in the university's possession. This included information regarding the student's educational background, age, attendance, engagement with online self-directed learning, and marks achieved. However, as the uniqueness of this cohort and the context in which it was studied was considered of primary importance, this is a limitation that this study needed to accept.

Conclusion

The study findings provide insight into undergraduate second-year students' assessment performance during a pandemic. Age, cross-medium engagement, and preparation were all shown to have an impact on marks. Those who engaged with taught sessions, online supported learning and the Skills for Practice website achieved higher mean marks than those who engaged less. These findings have the potential to influence how higher education institutions prepare students during intense periods of change, and this has relevance, as the face of contemporary education is ever-changing with the introduction of artificial intelligence, advances in immersive technologies and the constant changes in pedagogical practices.

Academics, therefore, need to continually review their practice, and also the factors that influence students' ability, and to drive to engage in the various teaching mediums. The findings from this study suggest that all parts of the blended learning package are equally important.

KEY POINTS

• Students disengaged with different aspects of blended learning during the COVID-19 pandemic
• Engagement with all aspects of blended learning led to significantly higher performance
• Students with disability support recommendations achieved significantly lower results than those without during the period of blended learning
• Significant differences in performance between these groups resolved as teaching returned to campus, and following the introduction of additional academic writing support

CPD reflective questions

• How can online learning (both taught and directed) be made more engaging for students?
• What can be done to better support students with disabilities when they are unable to attend campus?
• To what extent do undergraduate nursing students need direct support to develop academic writing skills?