For some time, how we assess and validate scholarly contributions has been a topic of intense debate and scrutiny within the evolving landscape of academic research.
The San Francisco Declaration on Research Assessment (DORA) emerged as a pivotal initiative within this conversation. It challenged the longstanding reliance on flawed metrics such as the Journal Impact Factor (JIF) and instead champions a more holistic, transparent, and equitable approach to evaluating research.
This article investigates the DORA movement’s origins, core principles, and far-reaching implications. Drawing insights from a diverse set of reference materials, it explores how DORA has been driving institutional change, reshaping research assessment practices, and empowering researchers to showcase the true value of their work.
Along the way, it will uncover the key challenges, ongoing efforts, and future directions of this initiative in order to help us all develop a deeper understanding of the DORA landscape.
The Birth of DORA: Addressing the Misuse of Journal Impact Factors
The San Francisco Declaration on Research Assessment was born out of a growing concern among scientists and journal editors about the overreliance on the Journal Impact Factor (JIF) as a proxy for the quality and impact of individual research articles. Initially developed as a tool for librarians to manage journal subscriptions, this metric was increasingly misused in hiring, tenure, and funding decisions, often to the detriment of researchers and the advancement of science.
In December 2012, a group of researchers and journal editors convened at the American Society for Cell Biology (ASCB) meeting in San Francisco, determined to address this pressing issue. They recognised that the JIF, with its inherent flaws and limitations, was distorting the scientific landscape, leading to a singular pursuit of publishing in a select few high-impact journals, often at the expense of timely and impactful research communication.
The Core Principles of DORA
The DORA declaration, which has since been signed by over 24,000 individuals and 400 organisations worldwide, outlines a set of core principles and recommendations aimed at transforming research assessment practices. At the heart of DORA is the fundamental tenet that ‘the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published’. [1]
Key DORA recommendations include:
- Avoiding the use of journal-based metrics: DORA explicitly calls for institutions, funding agencies, and publishers to refrain from using journal-based metrics, such as the JIF, as a substitute measure of the quality of individual research articles or to assess an individual scientist’s contributions.
- Considering a broader range of research outputs: DORA encourages the evaluation of a diverse array of research outputs, including datasets, software, patents, and other scholarly contributions, in addition to traditional publications.
- Emphasising qualitative indicators of impact: DORA advocates for incorporating qualitative measures of research impact, such as influence on policy, practice, and societal well-being, alongside quantitative metrics.
- Empowering Researchers: DORA empowers researchers to actively challenge inappropriate research assessment practices and promote best practices that focus on the value and influence of their specific research outputs.
Driving Institutional Change: DORA Signatories and Adoption Strategies
The growing momentum of the DORA movement led to a significant shift in the research assessment landscape, with institutions, funding agencies, and publishers worldwide embracing its principles and implementing new policies and practices.
Institutional Signatory Highlights
Some notable DORA signatories include:
- Major research universities: The University of Cambridge, Harvard University, and the University of California system have all signed DORA, committing to its recommendations.
- Funding agencies: Prominent funders such as the National Institutes of Health (NIH), the Wellcome Trust, and the European Research Council have endorsed DORA, pledging to align their assessment practices accordingly.
- Publishers: Leading academic publishers, including PLOS, eLife, Springer Nature, Taylor & Francis, and OUP, have adopted DORA principles, distancing themselves from the misuse of the JIF.
Adoption Strategies and Best Practices
As institutions navigate the implementation of DORA, a range of strategies and best practices have emerged:
- Transparent hiring and promotion criteria: Many universities have explicitly stated that the scientific content of a researcher’s work, rather than publication metrics or journal prestige, will be the primary focus in hiring, tenure, and promotion decisions.
- Diversifying research outputs: Institutions recognise a broader spectrum of research contributions, from datasets and software to policy briefs and community engagement, in their assessment processes.
- Developing qualitative evaluation frameworks: Some organisations have introduced innovative qualitative assessment tools, such as narrative CVs and impact case studies, to capture the multifaceted influence of research.
- Providing guidance and training: Universities and funding agencies offer resources and workshops to help researchers understand and implement DORA principles in their practices and communications.
The movement has also significantly impacted the peer review process and the broader landscape of scholarly communication.
Rethinking the Role of Peer Review
DORA sparked a critical examination of the peer review system, challenging the notion that it is a foolproof mechanism for ensuring research quality. The declaration acknowledged that peer review, while essential, is not a direct test of truth but rather a measure of plausibility. This recognition led to calls for reforms that enhance transparency, reduce bias, and better align peer review with DORA’s core principles.
Embracing New Communication Channels
Parallel to the DORA movement, the rise of preprint servers, such as bioRxiv and arXiv, enabled researchers to share their work more rapidly and efficiently, often before formal peer review. This shift empowered researchers to take control of their communication channels, reducing the reliance on high-impact journals and the associated delays in disseminating research findings.
Overcoming Challenges and Addressing Concerns
While the DORA movement gained significant traction, it also faced various challenges and concerns that must now be addressed to ensure its long-term success.
Concerns about Discrimination and Bias
One of DORA’s primary criticisms is the potential for increased discrimination and bias in research assessment, particularly in cases where hiring and funding committees are tasked with evaluating research outputs across diverse fields. The concern is that a lack of specialised expertise could lead to privileging familiar research topics and marginalising less familiar areas.
To address this issue, DORA advocates for developing rigorous, transparent, and accountable assessment processes that mitigate the influence of individual biases and ensure equitable evaluation.
Balancing Quantitative and Qualitative Measures
Another challenge lies in striking the right balance between quantitative and qualitative measures of research impact. While DORA emphasises the importance of considering a broader range of impact indicators, there are concerns about the feasibility and consistency of implementing such qualitative assessments across institutions and disciplines.
Ongoing efforts focus on developing standardised frameworks and best practices for effectively integrating quantitative and qualitative measures, ensuring that research assessment captures the multifaceted nature of scholarly contributions.
Resistance to Change
In some quarters, the deep-rooted reliance on traditional research assessment practices, such as using the JIF, led to resistance to the DORA principles. Overcoming this inertia requires sustained advocacy, education, and the demonstration of tangible benefits to researchers, institutions, and the broader scientific community.
The Future of DORA: Towards Responsible Research Assessment
As the DORA movement continues to evolve, the future of research assessment holds promising developments and ongoing challenges.
Expanding Reach and Influence
The DORA declaration gained significant global traction, with signatories spanning diverse disciplines and regions. The initiative aims to expand its reach further, engaging with a wider range of stakeholders, including policymakers, research administrators, and the general public, to drive systemic change.
Developing Robust Evaluation Frameworks
Ongoing efforts focus on developing robust, discipline-specific evaluation frameworks incorporating DORA principles. This includes creating standardised tools, guidelines, and best practices to assist institutions in implementing responsible research assessment practices.
Fostering Collaborative Partnerships
The DORA movement recognises the importance of collaborative partnerships within and across institutions to share knowledge, resources, and best practices. This collaborative approach aims to accelerate the adoption of DORA principles and ensure their consistent application across the research ecosystem.
Addressing Emerging Challenges
As the research landscape continues to evolve, the DORA initiative must remain agile and responsive to emerging challenges, such as the rise of artificial intelligence in research, the growing emphasis on open science, and the need to address systemic inequities in academia.
Final thoughts…
The San Francisco Declaration on Research Assessment emerged as a central pillar in transforming research assessment practices worldwide. By challenging the overreliance on flawed metrics, championing a more holistic and equitable approach to evaluating scholarly contributions, and empowering researchers to take control of their narratives, DORA can reshape the foundations of academic success and progress.
As the DORA movement continues to gain momentum, it is incumbent upon all stakeholders – researchers, institutions, funding agencies, and publishers – to embrace its principles and work collaboratively to build a research ecosystem that truly values the quality, impact, and diversity of scholarly endeavours. By doing so, we can unlock the true potential of research and ensure that the advancement of knowledge is driven by merit, innovation, and the pursuit of excellence.
Peer Review Management Services by PA EDitorial
PA EDitorial specialises in providing comprehensive peer review management services to publishers and editors, enabling them to introduce new workflows or optimise existing ones. Our expertise lies in alleviating the managerial burdens of peer review, allowing editors, authors, and academics to dedicate their time to developing, researching, writing, and evaluating innovative ideas that contribute to the advancement of scientific and academic knowledge worldwide.
Key Features of PA EDitorial’s Services
- Streamlined workflows: PA EDitorial assists in streamlining the peer review process, enhancing efficiency and effectiveness in academic publishing.
- Managerial support: We take on the managerial responsibilities of peer review, allowing editors, authors, and academics to focus on their core work.
- Innovation facilitation: By managing the peer review process, PA EDitorial supports the exploration and assessment of groundbreaking research, contributing to the enrichment of global academic and scientific knowledge.
- Professional collaboration: Working closely with publishers and editors, we ensure the implementation of tailored solutions to meet specific needs and improve overall peer review practices.
PA EDitorial’s dedication to excellence in peer review management is evident in its commitment to supporting the advancement of academic and scientific endeavours.
Learn more about PA EDitorial.
[1] https://sfdora.org/read