The entire peer review process is one of questioning. This isn’t because a peer reviewer is required to be a relentless sceptic or a pathological doubter. It’s because the exercise is extraordinarily demanding.
A peer review isn’t like the review of a book or a film, in which the consequence may, in some cases, be economically significant but most of the time immaterial. Instead, when asked to peer review a new paper, the academic assigned to the task assumes an enormous responsibility. Their job is to poke, test and probe, shine lights into every corner, subjecting the research, the argument and the conclusion to the most rigorous of assessments. That’s the nature of academic publishing, and it isn’t a casual business. On the contrary, it is disciplined, forensic and exhaustive.
Questions are the basis of every step in the procedure. Before reviewers even accept a commission, they have to ask questions of themselves. Are they qualified? Is their expertise and experience appropriate? These are questions of competency. There are also questions of practicality. Do they have the time to devote to a thorough review, and can they meet the editors’ deadlines?
Then there are other matters to determine, which are more about ethical concerns. For example, does the reviewer have any competing interests – financial, professional or even personal – which should be disclosed to the editors of the journals involved and are they prohibitive enough to rule out involvement?
Once the reviewer is satisfied that they may proceed with a clear conscience, a feasible time frame and the necessary academic skills, the questioning continues throughout the review itself.
Identifying the audience
No paper exists in a vacuum. Its purpose is to demonstrate, educate, provoke and convince, yet it is aimed at a specific group of people. Its intended audience should be clear on the face of it, but the reviewer will also ask whether the topic is clear and currently relevant, with appropriate vocabulary and a sufficiently comprehensive introduction. If the paper comes with its own summary, does that match the reviewer’s reading of the main body, or does it need adjustment? Also, thinking again in terms of the intended audience, does it make reasonable assumptions about the level of knowledge required to follow and appreciate the argument?
The coherence of the argument
The first requirement of any academic or scientific paper is that it be original. The second is that it should make a valuable contribution to its field of study. These are essential questions, but a peer reviewer will dig deeper to determine whether the argument is specific and clear enough, however complex. Is the thesis statement adequately expressed, and, crucially, does the paper make a watertight case for its own significance? If there is any doubt that the thesis is of negligible importance, this represents a serious weakness at its heart. Equally, the reviewer will ask whether there are any contradictions or digressions and, even more importantly, if those contradictions and digressions are justified by what they add to the paper. Complex arguments sometimes contain what appear to be anomalies, so it’s important to ask whether they strengthen or undermine the case being made.
Results and conclusions
The conclusions of any paper shouldn’t be taken for granted or accepted as proven without further rigorous questioning. Are the results presented clearly and accurately? Does the paper contain all relevant data, or are there obvious holes? Are any gaps or limitations acknowledged with appropriate caveats? Are the conclusions developed in terms of their broadest calculable implications?
Stylistic concerns
Academic and scientific papers can be notoriously dry. To some extent, this is both desirable and inevitable. An element of restraint is vital in reporting even the most ground-breaking or world-changing research. However, a peer reviewer will be conscious of how important it is for a paper to be more than simply readable and apply the appropriate stylistic editorial guidance. Is it both authoritative and accessible? Is the tone suitable and the phrasing not excessively elaborate? Does it thoroughly expound its aims, methods, results and conclusions? Is it grammatically and syntactically correct, and, ultimately, is it an enjoyable, satisfying read?
References and methodology
The argument and conclusions of any paper will stand or fall on the quality of its methodology and the authority of its references. This will always be a significant part of any peer review because it looks beyond the words and the thesis – at the research infrastructure on which the entire enterprise rests. Are the chosen sources appropriate for the questions being addressed? Do the references reveal any potential conflict or bias? Does the author let unsupported assumptions slip through their intellectual filter? Are there any apparent authorities missing that a member of the target audience would notice and question? Are the majority of sources primary ones, and where these are not used, is there adequate justification for relying on secondary ones? Have the proper controls and validation methods been used?
This is not an exhaustive list; it is virtually impossible to draw up such a thing. Nevertheless, these are all highly relevant questions that any peer reviewer would ask. However, every paper is an individual piece of work, and no prescriptive set of rules can be expected to cover all eventualities. The general lesson is that a peer reviewer will take nothing as proven until they see the evidence in the paper, presented in conjunction with the references cited.
There is another final question to ask when all the review work might seem to have been done.
Once the reviewer has read, digested and taken apart the paper, they should ask themselves: do they have any questions? By this, we mean the reviewer must feel confident in the thoroughness and professionalism of the paper. If there is any suspicion of incompleteness or sense that something has been assumed without proof or dismissed without consideration, this must be noted.
If, after all the questioning and analysis, the reviewer remains unconvinced, then this will need to be explained in the review, with clear reasons for the lingering doubt.
Peer review is not something to be treated lightly. Conducted poorly, it can undermine or even damage the field of study, which is the focus of the paper under review. On the other hand, carried out in a remorseless spirit of questioning, the review itself can significantly contribute to the sum of academic and scientific knowledge.
If you would like to know more about how PA EDitorial or EDiTech can help you, please feel free to contact us at info@paeditorial.co.uk.
PA EDitorial – No task too big or too small