Andrew Low, University of Liverpool

Awarding gaps—persistent disparities in academic outcomes between socio-demographic groups—are a key metric used to assess equity of outcomes in STEM education. Despite growing awareness, many methods used to measure and track these gaps overlook the role of confounding factors and the hierarchical structure of institutional data. Drawing on insights from a recent study of first-class STEM degree outcomes at a Russell Group university (Low & Kalender, 2024), this article highlights the importance of robust statistical approaches to identifying and tracking these gaps.

The Pitfalls of Simplicity: Why Basic Approaches Fall Short

Common methods for quantifying awarding gaps, such as comparing raw proportions or mean values, are widely used but can be misleading. In our study, a simple comparison of male and female students achieving first-class STEM degrees suggested that male students outperformed females, with female students having 0.85 times the odds (95% CI: 0.78–0.92) and an average 4% lower probability of achieving a first-class degree. However, this univariate analysis did not account for confounding factors—such as pre-university attainment, ethnicity, disability, and socioeconomic status—or the hierarchical clustering of students within STEM subjects. After adjusting for these factors, the findings reversed: female students had 1.40 times the odds (95% CI: 1.25–1.58) and an average 7% higher probability of achieving a first-class degree (Figure 1). So, what explains this dramatic shift?

The Role of Hierarchical Models in Understanding Awarding Gaps

Hierarchical models account for clustering within data and offer a more nuanced approach to analysis (Van Dusen & Nissen, 2019). For example, students are nested within subjects, which are further nested within faculties and institutions. By allowing the intercept in our hierarchical logistic model to vary by STEM subject, we accounted for baseline differences in the awarding of first-class degrees across subjects (Figure 2). Our analysis revealed substantial subject-level variability. Physics and Computer Science, for example, had high first-class attainment rates (50.0% and 49.5%) but were male-dominated, with male-to-female ratios of 3.27 and 7.21 respectively. In contrast, Psychology, with a much lower first-class attainment rate (21.5%), had a male-to-female ratio of 0.17. Without adjusting for these differences, comparisons based solely on raw proportions across the dataset suggested that male students were more likely to achieve first-class degrees. This is because male-dominated subjects with high first-class rates disproportionately influenced the overall results, while female-dominated subjects with lower attainment rates skewed the data against female students. However, when we accounted for subject-level variability, we found that, under comparable conditions, female students were actually more likely than male students to achieve a first-class degree. This demonstrates the critical importance of adjusting for nested data structures to avoid misleading conclusions when conducting awarding gap analysis.

Beyond Binary Metrics: Toward Nuanced and Intersectional Approaches

Intersectional analyses, which consider overlapping identities, are essential for uncovering nuanced disparities. By including interaction terms in hierarchical logistic models, we can investigate how the effect of one variable changes in the presence of another. For example, our analysis highlights that ethnicity-related disadvantages in achieving first-class STEM degrees vary by gender. Among male students, those from minority ethnic backgrounds are significantly less likely to achieve first-class degrees compared to their White male peers, with an average 8% lower probability (AME = 0.08, 95% CI: 0.05-0.12). For female students, this disadvantage is even greater: minority ethnic female students have an average 15% lower probability of achieving first-class degrees compared to White female students (AME = 0.15, 95% CI: 0.11-0.20). These findings underscore how the intersection of gender and ethnicity compounds inequities in first-class STEM degree outcomes (Figure 3). These insights reinforce the importance of intersectional approaches for developing a deeper understanding of awarding gaps. To advance this research, we recommend moving beyond binary metrics. Future studies should consider hierarchical ordinal regression, which models all possible degree outcomes while accounting for data clustering. This aligns with recent calls for methodological advancement in STEM equity research (Hubbard, 2024).

Awarding Gaps Are Not Static: Tracking Changes Over Time

Including interaction terms between socio-demographic variables and year of graduation in hierarchical models allows for the tracking of awarding gaps over time, offering essential insights into trends and the effectiveness of interventions. Figure 4 highlights the key trends observed in our study:

• Persistent Ethnicity Gaps: Black and Asian students consistently had lower probabilities of achieving first-class degrees compared to their White peers, with no statistically significant change observed between 2013/14 and 2021/22.
• Shifting Gender Dynamics: The historically higher probability of female students achieving first-class degrees has decreased over time, raising important questions about changes in institutional practices and policies.
• Pandemic Effects: During the COVID-19 pandemic, changes in assessment methods led to an overall increase in the proportion of first-class degrees awarded. There are also indications that some awarding gaps may have narrowed in 2020/21.

Implications for Physics Education Research

Addressing awarding gaps requires robust statistical methods that account for confounding factors, hierarchical structures, and intersectional identities. Simplistic approaches to analysing these gaps risk leading to ineffective or misdirected interventions. By leveraging advanced models and tracking trends over time, institutions can design evidence-based, targeted strategies while monitoring their impact. As a discipline rooted in precision and scientific rigor, physics must apply these principles to the study of awarding gaps.

References

1. Low, A. M., & Kalender, Z. Y. (2024). Predictors and Socio-Demographic Disparities in STEM Degree Outcomes: A UK Longitudinal Study using Hierarchical Logistic Regression. arXiv preprint, arXiv:2408.05853. https://doi.org/10.48550/arXiv.2408.05853

2. Van Dusen, B., & Nissen, J. (2019). Modernizing use of regression models in physics education research: A review of hierarchical linear modeling. Physical Review Physics Education Research, 15(2), 020108. https://doi.org/10.1103/PhysRevPhysEducRes.15.020108
   
3. Hubbard, K.E., Institution level awarding gap metrics for identifying educational inequity: useful tools or reductive distractions?, Higher Education 10.1007/s10734-024-01216-y (2024). https://link.springer.com/article/10.1007/s10734-024-01216-y