In search of SDGs in REF Impact Case Studies

In May 2020 we released our report, Contextualizing Sustainable Development Research, showcasing the growth in research around the UN’s Sustainable Development Goals (SDGs). We are following last month’s blog post on the UK’s research contributions to the UN SDGs observed through national assessment exercises with more analysis from Dr Juergen Wastl, Dr Briony Fane and Bo Alroe from Digital Science’s Consultancy Team. In this post, they focus on the impact of the UK’s research excellence framework (REF) submissions in relation to the UN SDGs.

Juergen Wastl is Director of Academic Relations and Consultancy at Digital Science. He previously headed up the Research Information team at the University of Cambridge’s Research Strategy Office and worked for BASF managing BMBF-funded projects internationally.

Briony Fane is a Research Analyst at Digital Science. She has a higher education background, having gained a PhD from City, University of London, and has worked as both a researcher and a research manager.

Bo Alroe has worked with research management and administration since 2004,  and currently as Director of Strategy with Digital Science. Bo is from Aalborg, Denmark, where he studied and currently lives with his family.

Introduction

In this analysis, we have used Dimensions on Google BigQuery to analyse the Impact Case Studies from REF 2014, the first exercise to evaluate the impact of research outside of academia. Impact Case Studies form an important part of all university REF submissions, with each describing both the research and its impact beyond academia. The inclusion of impact as a component of the REF reflects a growing interest in demonstrating the value of academic research in society.

Of the 6,975 Impact Case Studies submitted to REF 2014, 6,737 were included in the publicly available HEFCE dataset. Each case study has a unique digital object identifier (DOIs). We have taken these DOIs and used Dimensions on Google BigQuery to connect the REF Impact Case Studies to SDG categories using Dimensions’ SDG classification scheme launched last year.

Using Google BigQuery to connect external datasets to Dimensions to gain important insights

Thanks to HEFCE’s (now Research England) freely available database of REF research submissions, we were able to merge this with Dimensions. By applying Dimensions’ SDG filter it was easy to ascertain the proportion of research output associated with sustainable development submitted to REF 2014 underpinning the Impact Case Studies. We were also able to determine other trends within this information, including ‘types of impact’ associated with the case studies using publicly available code in the Dimensions BigQuery Lab; these include technological, societal, environmental, political, legal, cultural, health and economic types of impact, as defined in the post-REF report we published with King’s College London and HEFCE (now Research England).

In this case, we were able to leverage Dimensions data by enriching it with additional information in the form of impact case study data via Google BigQuery. This levels the playing field across databases, allowing users to carry out informative analysis and gain a greater depth of understanding of REF Impact data. This methodology provided the basis for the following two analyses.

Prevalence of SDG-related research underpinning Impact Case Studies

By analysing the number of SDG-related research publications from total publications between 1993 to 2013 (the two decades before REF 2014 submission) and the number and percentage submitted to the ‘output’ (REF2) and ‘impact’ (REF3) elements of REF 2014, the data show that SDG-related research was significant before the UN SDG goals were introduced in 2015. We see that proportionately more research with a sustainable development focus was submitted to demonstrate impact in REF 2014 than for research outputs. Interestingly, there was 25% overlap of impact case study research publication references with outputs submitted to REF2 in 2014¹.

Alignment of SDGs within the Dimensions categorisation and REF-related ‘types of impact’

SDGs can be clustered into three overarching pillars: Environmental, Economic and Social. The Stockholm Resilience Institute³ has assembled the SDGs by these overarching groups in a diagram that resembles a wedding cake (with Social labelled Biosphere here). This hierarchical aggregation of the individual goals allows us to better visualise and analyse overall trends, and enables comparisons with the REF Main Panel structure (A-D) to retrieve a high level view of sustainable development goals in REF Impact.

Underpinning research categorised by the HEFCE ‘impact types’ and by the three pillars of SDGs as outlined above nicely validates the outcome of the SDG classification system in Dimensions. Table 2 reveals that 62% of underpinning research classified within SDG13, Climate Action, correlates with HEFCE’s Environmental impact type. 45% of research classified as SDG3 Good Health and Well-Being correlates with HEFCE’s Health impact type, and 47% of research associated with SDG16 Peace, Justice and Strong Institutions correlates with HEFCE’s Societal impact type.

As we have already noted, the time period for REF 2014 precedes the formalisation of the UN SDGs but, as we demonstrated in our last blog, UK research exhibits a high proportion of SDG-tagged publications. This is even more pronounced in the underpinning research accompanying the REF Impact Case Studies, showcasing the existing contribution of research in the context of sustainable development to both ‘Impact’ and ‘Excellence’ in the REF 2014 submission.

Comparing citation averages for SDG-related underpinning research versus non SDG-related underpinning research

REF Impact Case Studies were categorised into eight types by HEFCE (now Research England): Cultural, Economic, Environmental, Health, Legal, Political, Societal and Technological. Figure 2 outlines the citation averages for underpinning research publications categorised by impact type that have been SDG-tagged in Dimensions, versus those not SDG-tagged. In particular, we see that citation averages are twice as high for SDG-related publications associated with Economic and Environmental impact types than those that are not SDG-related. Conversely, citation averages for underpinning research outputs associated with Technological and Cultural impact types fall below the citation average of those outputs that are not associated with SDGs. It could be that, for both Environmental and Economic impact types, there is an existing awareness of SDG-related sustainable development thinking, particularly environmental impact; we know that climate change didn’t emerge in 2015 with the introduction of the SDGs. For research related to Technological impact, the focus may have been on more non-sustainable development related technologies.

Looking ahead, we might expect that research underpinning Impact Case Studies in REF 2021 will see a notable increase in SDG-related publications as we continue the UN’s decade of action for realising the sustainable development goals.

Weaving the four REF Main Panels into the story provides us with a slightly different insight into the contribution that the overarching REF Panels made to sustainable development research in REF 2014, as seen in Figure 3:

• Main Panel C (social sciences) contributes most and is evident in all SDGs across all HEFCE impact types. This Panel contains the greatest quantity of underpinning research publications associated with Impact Case Studies overall and confirms the contribution of this Panel to REF2 outputs
• Unsurprisingly, Main Panel A (medicine, health and life sciences) contributes predominantly through SDG3 Health and Well-Being, and across to HEFCE’s Health impact type related Impact Case Studies
• Main Panel B (physical sciences, engineering and maths) mostly contributes to SDG7 Affordable and Clean Energy, and SDG13 Climate Action through to Technology and Environmental ‘impact types’
• Finally, Main Panel D (arts and humanities) which accommodates a more diverse portfolio of underpinning research associated with SDGs, but dominated by SDG16 associated research, does not solely contribute to cultural impact type categorised case studies as might be expected

Research in the context of sustainable development (using Dimensions SDG Classification scheme) acts as a pivot between HEFCE’s ‘Impact Type’ and the four REF Main Panels, showcasing the diverse interaction between the underpinning research in the Impact Case Studies component of REF 2014, its discipline and contribution to the UN SDGs, even before these were formally implemented.

Conclusion

This blog has revealed a starting point for us looking at the scope for interpreting SDGs through the lens of Impact. Utilising the reference lists submitted with the case studies provided us with a context within which to explore research associated with sustainable development in REF 2014, ahead of the formalisation of the UN SDGs in 2015.

Using Google BigQuery allows us to draw interesting conclusions about the impact of research submitted to REF 2014. By integrating Dimensions data with a relevant external dataset we have been able to showcase the potential for future application in developing insights on and beyond sustainable development with impact case studies from the forthcoming REF 2021.

---

Footnotes

1: The document can be viewed here

2: Each underpinning research publication may be associated with more than one SDG classification. The total number of publications we are working with is 21,829 but, with the overlap of SDG- and UoA-associations, these 21,829 publications are represented 32,684 times

3: The SDG ‘wedding cake’, developed at the Stockholm Resilience Centre, Stockholm University by the Centre’s science director Carl Folke