Academic Psychiatry 28:81-87, June 2004
© 2004 Academic Psychiatry
How to Review a Manuscript: A "Down-to-Earth" Approach
Laura Weiss Roberts, M.D., M.A.,
John Coverdale, M.D., F.R.A.N.Z.C.P.,
Kristin Edenharder, B.A. and
Alan Louie, M.D.
Dr. Roberts is the Editor-in-Chief of Academic Psychiatry. Drs. Coverdale and Louie are the Associate Editors for Academic Psychiatry. Ms. Edenharder is the Assistant to the Editors.
A thoughtful review of a manuscript is a gift. It is a gift of expertise, of time, and of careful consideration. It is a gift for authors, editors, and the field. Performing a thoughtful peer evaluation of a manuscript is an acquired skill, however, and one that may not be formally taught in many academic departments or postgraduate training programs. The aim of this short piece is to provide some "down-to-earth" guidance for early career faculty, prospective authors, and peer reviewers on how to write a manuscript review. This represents a fundamental skill in academics, and this paper is the first in a series that will appear in our journal providing "down-to-earth" guidance related to academic skills for our readers.
Peer Review and Editorial Decision-Making
The goals of peer review are to assist the editors in forming a decision concerning publication of a manuscript and to provide constructive feedback to authors in order to enhance the quality of the final written product.
Figure 1 depicts the editorial and publication process common to most journals. Once a manuscript is received, it is read and "triaged" by the Editors. Although most journals handle this initial evaluation of papers similarly (i.e., the papers are read by the editors and/or editorial assistants), different journals have different thresholds at this stage of manuscript consideration. Some journals are quick to reject papers at this stage, whereas others (like Academic Psychiatry) seek to include all papers that have a chance of success. Our journal sends out for peer review the vast majority of manuscripts we receive because of our commitment to helping authors. We believe that the peer review process is inherently valuable, even if it is decided that an individual manuscript will not be published. For Academic Psychiatry, the main reason for rejection at this early point would be an inappropriate topic, i.e., the goals of the manuscript would be seen as falling outside of the mission of our journal.
Once the decision is made to send a manuscript out for review, up to five potential reviewers are contacted via e-mail to inquire about their willingness to review a manuscript within a 3-week period. This time frame is necessary so that the review process and editorial decision-making can occur in a timely fashion for authors. An editorial assistant tracks the progress of each review and will send reviewers periodic "reminder" e-mails once a review's optimal deadline has been reached. If a reviewer commits to completing a review, it is important that the reviewer follow through with this commitment. Likewise, if a reviewer is unable to review a particular manuscript, it is important to communicate this to the editorial staff. We will then reassess the number of willing reviewers for each manuscript and invite new reviewer candidates as necessary. As the editors will often wait to receive all invited reviews before making a decision, a late review or an unanswered invitation to review may cause substantial delays in the peer review process.
Comments received back from peer reviewers can be brief (1–2 paragraphs) or lengthy (up to 3 pages). Reviews usually identify a range of strengths and weaknesses found in the manuscript, and the reviewers provide various ratings on publishability and acceptability. The editors are responsible for using the reviews to inform their editorial decision. Ordinarily we will not render a decision unless we have received a minimum of two reviews for a given manuscript. This policy ensures that authors are given a substantial amount of specific feedback from which to draw in drafting a revision. Often reviews are mixed (i.e., a single reviewer will have an ambiguous response to a paper) or in conflict (i.e., different reviewers express different degrees of enthusiasm—or not—for a paper).
Once a decision has been reached on a manuscript, the authors receive a letter or e-mail from the editor(s), accompanied by reviewers' comments, informing them of this decision. With the transition to our new online submission system, more and more communication from Academic Psychiatry's main editorial office is handled electronically. E-mail allows for quick and efficient contact, which is beneficial to all. Decisions on manuscripts range from "accepted" (with or without a requirement for minor revisions) to "rejection." Most commonly, we defer making a final determination on a manuscript at this point—again, this provides an opportunity for authors to pursue a revision. A deferred decision should not be perceived as a rejection. Should a decision on publication be deferred, authors may rework their paper and resubmit for editorial re-review.
When a paper is revised and resubmitted, the editors read this next version and then there are several possible outcomes, including acceptance, rejection, or re-review. We typically ask for review of the revision by the original reviewers. Further revision and iteration of the preceding process may again be suggested. For these reasons, providing a fair review and editorial decision can be a protracted process. It is also for these reasons that an expeditious contribution from the reviewers is appreciated by editors and authors.
Elements of Peer Review
The elements of peer review include "comments to the author(s)" and a rating form including "confidential comments to the editors." The "comments to the author(s)" are sent to the authors, with, in the case of Academic Psychiatry, the reviewers' identities masked.
The most helpful review is one that articulates the strengths of a paper while also assiduously identifying the limitations of the manuscript that can be addressed in a revision. Nevertheless, even if a paper is well received overall, reviewer comments on manuscripts are commonly "negative" (1). Authors naturally find this discouraging, and we at Academic Psychiatry make every effort to encourage reviewers to identify positive aspects of the papers we receive. In addition, while critical comments of a constructive nature are solicited, our reviewers are actively discouraged from using unduly harsh or pejorative language.
The "comments to the editors" form used by Academic Psychiatry incorporates a checklist with a dichotomous (yes/no) scale, a global rating of overall publishability on a scale of 1 (unacceptable) to 10 (outstanding), and "confidential comments to the editors." The checklist presented on this form solicits information on originality, clarity of presentation, use of relevant references, length, clarity and adequacy of design, clarity of results, need for a statistical consultant, utility of tables and figures, and adequacy of discussion and conclusions. The global rating of overall publishability combines a rating of quality plus importance, with reviewers also asked to rate the publishability of the manuscript should it be revised in line with recommendations. Even though the editors decide the outcome of the peer review, the reviewers also check their recommendation, with possible responses including "accept as is," "modify with minor revisions," "modify with major revisions," or "reject." The comments to the editor generally provide a brief rationale for this recommendation. Based on the quality and importance of the research, the reviewer provides a succinct comment on the weight of strengths compared to weaknesses of the manuscript, and on the potential reversibility of significant weaknesses. The importance of a manuscript is judged in part by the strength of the results but also by whether it contains original ideas. For instance, less substantive data may be acceptable if the work is the first of its kind.
A "Down-to-Earth" Approach to Writing a Review
When a reviewer is approached by our editorial staff, he or she is given an abstract of the manuscript. There are several things to be considered when deciding whether or not to accept an invitation to review a manuscript. The reviewer might have some conflict of interest that prevents truly fair and objective review. Further, the topic or methodology may fall outside of the reviewer's ability to adequately appraise it. Finally, it may not be possible for the reviewer, given other time constraints or commitments, to complete the review within the time frame requested. In any of these cases, it is desirable for the reviewer to promptly request recusal. If a reviewer is unable to review a work, an editorial assistant may contact him or her to ask for suggestions for alternate reviewers. As experts in their field, reviewers can often recommend other qualified experts in that same area of study. In all cases, manuscripts released to reviewers are confidential, and their contents should not be discussed with others until they are published. A reviewer may invite a colleague to assist him or her with the review but must obtain prior approval from the editor(s) or the editorial assistant.
In approaching a review, it can be helpful to read the manuscript a first time before finalizing a response. Once time has passed and an initial impression is formed, the reviewer may return to the manuscript to finish the review. Reading the manuscript more than once, and coming back to it after a break of a day or more, even when the bulk of the review has been completed, enables a relatively fresh perspective. Depending on the type of article (e.g., reports, commentaries, scholarly reviews, brief reports), the review may require 1–3 hours (2) and 500 to 1,000 words.
At Academic Psychiatry, we advocate focusing on the goals of a manuscript at an early stage. The goals of the manuscript or the research hypothesis should be clearly specified and should fit with the mission of our journal. For an empirical paper, for example, it is important to appreciate that the research goals determine the methodology; a careful reading of the goals prepares one to review the Methods section, one of the most important sections.
The Methods should be evaluated for adequacy and clarity of the description of the methodological processes including design, procedures, ethical safeguards, and quantitative or qualitative analyses. Limitations in study design, like the absence of a control group or confounding factors, may reduce the validity of a study. The sample and method for sampling should be clearly described so that its representativeness to the population, to which the results will be generalized, can be assessed. One common problem in analytical research in education is the use of a small sample size, resulting in a lack of statistical power, such that even when true differences exist between groups, these are not detected. Information on any instruments used including validity (the extent to which an instrument measures what it purports to measure) and reliability (the extent to which an instrument provides consistent measurements) should be made available when possible. In essence, the Methods section should be sufficiently well described to allow someone else to accurately replicate the study.
The Results should be clearly presented and highlighted. Statistical tests should be appropriate to the question. The most important findings should be emphasized in a table, the less important in the text, but none in both places. The real-life significance of results should be judged; sometimes significant differences between groups exist only on a statistical basis and do not translate into practical differences. Some of the common weaknesses of manuscripts in medical education are inappropriate or incomplete statistics, omission of data, inconsistent or inaccurate data, and defective or unclear tables or figures (1). Reviewers who query their own ability to adequately assess the statistics should inform the editor of their concerns in case the opinion of a statistical consultant is needed.
The Discussion section should be judged in part by the ability of the author(s) to frame and interpret the main findings and to honestly assess strengths and limitations of the research. The reviewer is asked to judge whether the discussion and conclusions are adequately supported by the manuscript's findings or arguments. Overinterpretation of the data is a common weakness in medical education reports; speculative, unsubstantiated, or unsupported comments should be noted by the reviewer (1).
Paradoxically, the reviewer's last task is to assess the Introduction and References. The abstract should be assessed for its utility as a summary and for accuracy of content. The process of review is necessarily iterative in that the abstract can be fully evaluated only after the entire manuscript has been reviewed in detail (3). Reviewers judge the adequacy of the literature review, especially searching for the possibility that important references were omitted. Familiarity with the relevant literature is expected of the reviewer. Similar or contradictory studies should be exhaustively referenced; here the author(s) should be precise.
At the point of writing a review, it is helpful to go over the checklist of criteria provided by Academic Psychiatry on the "comments to editors" form in order to ensure that the review will be complete. This checklist is reviewed by the editors, but can also offer an excellent outline from which reviewers can formulate comments to the author(s). A common occurrence in reviews received by the editorial staff is that reviewers write all of their comments regarding the manuscript and potential improvements on the "comments to the editors" form, which is strictly confidential, and then write little or nothing on the "comments for author(s)" form. Many of these comments are helpful and insightful and would be beneficial to the author(s). If a reviewer would like his or her comments to be read by both editors and authors, it is best to include these comments on both forms.
In formulating comments for the author(s), the reviewer should first describe the intent and potential value of the manuscript, and note any strengths. Subsequent comments usually enumerate concerns, starting with the science (e.g., methodology, data analysis) and how these may be remedied if possible. The reviewer next turns to a critique of the Discussion, Introduction, and References. Lastly and optionally, some reviewers will include a few editing notes, pointing out grammatical, formatting, or typographical errors.
Potential Problems in Review
One mistake often made by reviewers is to assume that the manuscript will be rejected and to therefore provide little feedback to the author(s). Regardless of their recommendation on a manuscript, reviewers should embrace the collegial collaborative and teaching role entrusted to reviewers. We hope that reviewers will strive to provide constructive feedback that will aid authors in the future revision of their work. One cautionary note: the decision concerning publication ultimately belongs to the editors who often receive ambiguous and conflicting reviews. Reviewers should never predict the editorial decision in their "comments to the author(s)." In addition, we hope that reviewers will understand that their viewpoint is always carefully considered when editors deliberate over manuscript decisions. Even if a reviewer's recommendation about publishability does not line up with the ultimate decision by the editors, the contribution of the reviewer to the process is always sincerely appreciated and valued.
One challenge for the reviewer is to achieve a broad perspective on the potential contribution of a manuscript to the field. This is judged in part by the extent to which the science or scholarship achieves more than has been achieved before in our relatively new and developing field. The failure to appreciate the relative merits of a manuscript in this light will result in undue weight being assigned to perceived flaws.
Lastly, the reviewer must examine his or her potential biases (4). This includes the overrating or underrating of a study because the findings support or do not support a preexisting view of the reviewer or because it is published by a prominent or unrecognized author (4). Although data concerning the value of blinding reviewers to authors' identities suggests that the quality of review is not affected (5–7), Academic Psychiatry removes the authors' names and their affiliations from the reviewed manuscript in order to protect against some of these potential biases. Nevertheless, the responsibility of the reviewer is to recognize any biasing influences and to put them aside.
Peer Review and Academic Psychiatry
In an earlier column, one of us (LWR), wrote of the centrality of peer review to the functions of Academic Psychiatry and provided some resources for both writing and review (8). The Journal has an outstanding tradition of support and mentorship of prospective authors. We strongly encourage the efforts of those who are new or relatively new to authoring papers to get started and to contribute. We welcome queries about how to proceed, about the suitability of ideas for the Journal, and about relevant steps in the publication process. We are committed to professional development and to helping new authors attain skills of value to their academic achievement.
The tradition of Academic Psychiatry has emphasized an absolute respect, collegiality, and empathy in all interactions. It is an honor and privilege to be selected as reviewer and to have an opportunity to work cooperatively and constructively as teacher or mentor to the author. This is challenging, methodical, conscientious work worth doing truly well. We applaud the genuine commitment of our reviewers in this central role for the Journal, and we encourage new reviewers to embrace the opportunity to become involved in the peer review process. As editors, one of our responsibilities is to ensure that we have optimized our processes for reviewers and authors alike in advancing the missions of the Journal and the quality of publications. We welcome feedback from all about how we can perform these functions better.
We consider a careful evaluation of a manuscript by a knowledgeable and fair-minded peer as a critically important scholarly contribution. The "down-to-earth" process of peer review is invaluable for the continued success of authors and of journals. It assures the rigor and fosters the advancement of our field. Though challenging, reviewing manuscripts can be a gratifying duty in academics, a gift to our colleagues and a gift that we receive in return.
REFERENCES
- Bordage G: Reasons reviewers reject and accept manuscripts: the strengths and weaknesses in medical education reports. Acad Med 2001; 76:889–896[Medline]
- Hoppin FG: How I review an original scientific article. Am J Respir Crit Care Med 2002; 166:1019–1023[Free Full Text]
- Bordage G, Caelleigh AS: How to read "review criteria for research manuscripts." Acad Med 2001; 76:908–909
- Owen R: Reader bias. JAMA 1982; 247:2533–2534[Abstract/Free Full Text]
- Van Rooyen S, Godlee F, Evans S, Smith R, Black N: Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA 1998; 280:234–237[Abstract/Free Full Text]
- Godlee F, Gale CR, Martyn CN: Effect on the quality of peer review of blinding reviewers and asking them to sign their reports.: a randomized controlled trial. JAMA 1998; 280:237–240[Abstract/Free Full Text]
- AC, Cho MK, Winker MA, Berlin JA, Rennie D and the PEER investigators: Does masking author identity improve peer review quality? a randomized controlled trial. JAMA 1998; 280: 240–242
- Roberts LW: On the centrality of peer review. Acad Psychiatry 2002; 26:221–222[Free Full Text]
This article has been cited by other articles:
|
|
|
|
 
A. V. Neale, M. A. Bowman, and P. Lupo
Peer Reviewers for the Journal of the American Board of Family Medicine in 2008
J Am Board Fam Med,
March 1, 2009;
22(2):
102 - 106.
[Full Text]
[PDF]
|
|
|
|
|
|
|
M. L. Cooper
Problems, Pitfalls, and Promise in the Peer-Review Process: Commentary on Trafimow & Rice (2009)
Perspectives on Psychological Science,
January 1, 2009;
4(1):
84 - 90.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
J. H. Coverdale, L. Weiss Roberts, R. Balon, A. K. Louie, and E. V. Beresin
Improving the "Impact" of Academic Psychiatry
Acad Psychiatry,
May 1, 2008;
32(3):
169 - 172.
[Full Text]
[PDF]
|
|
|
|
|
|
|
R. G. Sheiman
The RSNA Reviewer Mentorship Program
Radiology,
September 1, 2007;
244(3):
631 - 632.
[Full Text]
[PDF]
|
|
|
|
|
|
|
A. V. Neale, K. L. Schwartz, and M. A. Bowman
Peer Reviewing for the Journal of the American Board of Family Medicine: What Does It Take?
J Am Board Fam Med,
November 1, 2006;
19(6):
643 - 647.
[Full Text]
[PDF]
|
|
|
|
|
|
|
J. Coverdale, L. Roberts, A. Louie, and E. Beresin
Writing the Methods
Acad Psychiatry,
October 1, 2006;
30(5):
361 - 364.
[Full Text]
[PDF]
|
|
|
|
|
|
|
K. Edenharder
Practical Knowledge for Academic Psychiatrists
Acad Psychiatry,
June 1, 2006;
30(3):
238 - 238.
[Full Text]
[PDF]
|
|
|
|
|
|
|
J. M. Provenzale and R. J. Stanley
A systematic guide to reviewing a manuscript.
J. Nucl. Med. Technol.,
June 1, 2006;
34(2):
92 - 99.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
J. M. Provenzale and R. J. Stanley
A Systematic Guide to Reviewing a Manuscript
Am. J. Roentgenol.,
October 1, 2005;
185(4):
848 - 854.
[Abstract]
[Full Text]
[PDF]
|
|
|
Get information about faster international access.
Privacy Policy
Copyright © 2004
Academic Psychiatry.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
