This report summarizes a workshop that the National Institute of Mental Health (NIMH) held on October 12, 2004, focused on statistical and methodological issues that often appear in the review of interventions research grant applications. Reviews are performed by a multidisciplinary group of reviewers that includes doctoral level clinical researchers who are familiar with the disorders and their treatments, as well as statisticians. Additional scientific experts and public participant reviewers, who represent various stakeholder perspectives, are recruited to ensure that each application gets a comprehensive, fair, expert, and balanced review.

Diverse perspectives in the review process add breadth and depth to the review, but statistical reviewers and clinical reviewers frequently have different perspectives that can add to the complexity of peer review. In addition, the nature of contemporary clinical research also adds to the complexity of the review process. Challenges in designing and conducting intervention studies include, for example, defining inclusion and exclusion criteria to select the target population; identifying promising and innovative treatments; selecting appropriate comparison conditions and designs that yield maximally interpretable, valid, and clinically useful results; implementing reliable, valid, and well-timed assessments to characterize intervention processes and outcomes; ensuring that the interventions are delivered with fidelity; and selecting and applying appropriate data-analytic techniques that will illuminate intervention effects and moderators/mediators of response. Issues involving the appropriateness of the proposed study’s aims, the design, and the planned data analysis, as well as the precision of fit among these elements of the application, are central issues within the review process.

The review process can seem mysterious to applicants who have not experienced it either in the form of receiving feedback from a review or participating in a review meeting. Thus, the purpose of this article is threefold:

1. To summarize points for consideration that were generated during the workshop and in subsequent discussions so that researchers who submit interventions applications can understand some of the design- and statistic-related components of review and can submit applications with a better understanding of the expectations of both clinical and statistical reviewers.

2. To de-mystify and make transparent some, but certainly not all, of the core issues that show up in review meetings where the clinical and statistical elements of an application are judged and weighed by reviewers who approach the review process with different perspectives.

3. To hopefully also serve as an enduring reference for future reviewers of intervention research applications.

The workshop summary is structured as a series of “points for consideration” formatted as questions, rather than as a list of explicit recommendations. The points reflect the participants’ opinions that the optimal content and level of detail for a given application depend on its research focus and scope of the proposed work. It should be emphasized that the workshop focused on issues relevant to the review of mental health interventions research applications and that the following points for consideration will not apply equally to all types of grant applications. The following points for consideration are intended as a potential resource for those who develop mental health interventions research applications and those who review them.

A central impetus for convening the workshop was to discuss how clinical research elements and design/analytic elements of applications should be considered and weighed in peer review. A clinically meaningful research question and sound methods and analyses are both necessary components of competitive applications. At times, however, there is a disconnection between or imbalance in the quality of these components. Extreme examples might involve compelling research questions paired with mismatched, underdeveloped, or seriously flawed analysis plans. Conversely, elegant analytic plans may be applied to less significant research questions.

At times, the review process identifies serious statistical and methodological problems, sometimes characterized by reviewers as “fatal flaws” that threaten the study’s ability to address the primary research question. These problems include but are not limited to selection effects, insufficient power in later-stage intervention studies, and differential attrition or other problematic sources of missing data that threaten the interpretability of the results. Many concerns identified in reviews represent more minor weaknesses that merely detract from the merit of an otherwise potentially viable application. In such circumstances, the overall evaluation of merit ultimately depends on the degree to which the application, in its current form, promises to yield meaningful results; the degree to which adjustments (e.g., different analytic strategies) could improve the application; and the degree to which the investigator team appears capable of addressing problems. Discussion regarding weighing the elements of the application is summarized in the following points for consideration:

Some studies are primarily aimed at advancing our understanding about the efficacy or effectiveness of an intervention and/or identifying moderators and mediators of response. Other studies that occur earlier in the research process are primarily focused on intervention development or novel adaptations of existing interventions and thus constitute pilot studies, where the goal is to assess feasibility, acceptability, safety, and tolerability (1). Depending on the type of study, and, accordingly, the funding mechanism, the nature of the statistical and methodological issues will vary (information about specific funding mechanisms is available at www.nimh.nih.gov). Relevant points for consideration are summarized below.

Much of the workshop discussion addressed aspects of the design and analysis portions of applications that impact whether the proposed study will be sufficient or optimal for addressing the research questions. Discussion focused on general considerations as well as specific topics, including strategies for minimizing and handling missing data, effect size considerations, and power analyses.

General Considerations

Missing Data

Missing data occur for a variety of reasons and affect both outcomes and covariates. The main consequence of missing data is potential bias in the estimate of the treatment effect; specifically, subjects for whom there is “completely” observed information may be systematically different from subjects with incompletely observed information. The workshop discussion acknowledged two fundamental facts regarding missing data: all studies are at risk for missing data and, moreover, the potential for bias arising from missing data exists even if the investigator proposes to undertake an intention-to-treat analysis. The following related points for consideration were noted:

Effect Size Considerations

An effect size is an index that measures the size of a treatment effect and is independent of sample size (11). Effect sizes are key components of treatment-related applications, as they may be used to represent the smallest effects that would be clinically meaningful.

Power Analyses

There are many aspects to a power analysis, but at the heart of any discussion of statistical power, a grant reviewer expects to find the explicit quantitative assumptions that are relevant to whether the proposed study design can answer the research question(s) of interest. A power analysis requires the explicit articulation of the specific outcome of interest and how it is quantified. Inherent in power considerations are a number of key questions: What is a clinically meaningful effect? What is the relation between the primary outcome measure and the research question of interest? How variable is the outcome of interest?

It was also noted that in addition to the scientific interest in the power analysis and sample size calculations, there is an ethical imperative associated with these analyses that should be considered, along with a host of other ethics issues in psychiatric research (17). Researchers have an ethical obligation to carefully consider the sample size proposed for a research study (18). In fact, the Committee on Professional Ethics of the American Statistical Association issued Ethical Guidelines for Statistical Practice states, “Avoid the use of excessive or inadequate numbers of research subjects by making informed recommendations for study size” (19).

Those informed recommendations are based on statistical power analyses, which integrate information from specific aims, hypotheses, preliminary studies, and methods (i.e., subjects, assessments, and data analytic procedures).

A key consideration in the evaluation of an application is whether the proposed statistical methods are appropriate in terms of the specific research questions and in terms of the data that will be collected (e.g., the amount of data, the nature and anticipated distribution of the data). Another consideration is the appropriate degree of complexity: competitive applications appropriately exploit modern analytic methods to yield maximally interpretable results and reveal robust effects. Perhaps ideally, the analyses would utilize the simplest methods likely to yield valid, precise, meaningful, and appropriately targeted results. The following points for consideration are potentially useful for avoiding methods that may be overly simple or strategies that entail considerably greater complexity than appears justifiable given the study design, aims, and data:

The integration of the clinical research and design and analyses aspects of interventions research is embodied in the collaboration between the clinical researchers and statistician collaborators who form the research team. The depth and quality of this collaboration is often evident, not just in the data analytic section of the application, but throughout (e.g., in the framing of the aims, in the data collection plan, and even in the budget justification). The following points for consideration reflect discussion regarding the statistician’s role:

The complex nature of contemporary clinical research necessitates bringing multiple viewpoints to bear in both the development and execution of research studies, and in the review process during which the application’s merit is evaluated. The purpose of the workshop and this summary was to address the common issues at the interface of clinical research and design/analytic elements of studies that pose challenges to researchers and to reviewers. Meaningful collaboration is clearly needed between experts who are familiar with the clinical elements of a disorder, existing treatments, and the needs of the health care field, and experts in design and analyses relevant to clinical trials research. This partnership is necessary to properly address clinically meaningful questions and identify studies that will yield interpretable results that can lead to improved treatment of mental and behavioral disorders.

Views expressed within this article represent those of the authors and are not intended to represent the position of NIMH, NIH, or DHHS. We thank the extramural scientists, public-participant reviewers, and NIMH Review and Program Staff who participated in this NIMH-sponsored workshop for their invaluable input. The authors also acknowledge the ongoing contributions of those who serve in the peer review process; their feedback was instrumental in conceptualizing the workshop and in the preparation of this report.

At the time of submission, Drs. Sherrill, Sommers, Arndt, Greenhouse, Guthrie, Normand, and Woolson disclosed no competing interests. Dr. Nierenberg and Dr. Phillips have provided full disclosure from several public and private sources that are available upon request. Dr. Leon served as a consultant to the FDA, NIMH, MedAvante, and Cyberonics during the past 12 months; he served on Data and Safety Monitoring Boards for Pfizer, Dainippon Sumitomo Pharma America, and Organon. Dr. Shear served on an advisory board for Forest Pharmaceuticals.