Advanced Search

REGULAR | May 01, 2009

An Institutional Postdoctoral Research Training Program: Predictors of Publication Rate and Federal Funding Success of Its Graduates

Randal G. Ross; Linda Greco-Sanders; Mark Laudenslager; Martin Reite

Academic Psychiatry 2009;33:234-240.

View Article Information

Received September 13, 2007; revised December 26, 2007, and February 21, 2008; accepted February 21, 2008. The authors are affiliated with the Department of Psychiatry at the University of Colorado Denver. Address correspondence to Randal G. Ross, M.D., University of Colorado Denver, Department of Psychiatry, Bldg. 500, Box F546, 13001 E 17th Pl, Aurora, CO 80045; randy.ross@ucdenver.edu (e-mail).

text A A A

Abstract

Objective: The National Institute of Mental Health funds institutional National Research Service Awards (NRSA) to provide postdoctoral research training. While peer-reviewed publications are the most common outcome measure utilized, there has been little discussion of how publications should be counted or what factors impact the long-term publication rates of trainees in these programs. Methods: The authors reviewed current curricula vitae from 92 graduates of an institutional NRSA and from the faculty mentors of that program to assess publications through 2005. Publications were weighted based on peer versus non-peer-reviewed and authorship position. Trainee and mentor factors were assessed for their impact on publication rates and on becoming principal investigators of larger scale federal grants such as a National Institutes of Health (NIH) R01. Results: Weighted publication scores correlate with total publication rates at such a high rate that the two scores can be used interchangeably. Forty-three percent of graduates average at least one publication per year after completing the postdoctoral program; 20% were listed as an independent investigator on a larger federal grant. The number of publications published during postdoctoral training and additional funded training beyond that provided by the institutional NRSA are correlated with increased posttraining program publication rates; other factors including gender, terminal degree, number of publications prior to postdoctoral training, and mentor variables had no significant impact. Additional funded training, male gender, and increased time since completion of the training are associated with increased likelihood of larger grant federal funding. Conclusion: Weighting publications by whether they were peer-reviewed and by authorship position appears to have little benefit over a simple counting of the number of publications. Publication during research training and the pursuit of funding for additional individual research training may be appropriate short-term goals within an institutional research training program.

Figures in this Article

A postdoctoral training experience is often considered a critical step in the career development of an independent researcher. The National Institutes of Mental Health (NIH) support both individual and institutional postdoctoral training programs (F32 Individual National Research Service Awards and T32 Institutional National Research Service Awards). In 2005, the National Institute of Mental Health (NIMH) supported training for 585 postdoctoral positions at a cost of over $38.3 million dollars (¹, ²). The majority of the support was provided via 135 institutional awards; 87% of NIMH-supported postdoctoral slots were via institutional awards. Despite this heavy investment in using institutional National Research Service Awards (NRSA) to fund postdoctoral research training positions, there is little literature on the effectiveness of using these federal research dollars.

One of the difficulties in assessing the effectiveness of postdoctoral training programs is a lack of consensus on how to define a successful outcome. One option would be to focus on the scientific impact of a program’s graduates. Hirsch (³) has suggested a method for assessing an individual’s scientific contributions. Often termed the h-index, this method attempts to quantify the impact of an individual scientist’s work. The h-index is calculated by determining the number of times each article is cited and then determining the highest number of articles with an equivalent number of citations. For example, an h-index of 7 indicates that an author has 7 or more publications and at least 7 of those publications have been cited 7 or more times; an h-index of 12 indicates that an author has 12 or more total publications and at least 12 of those publications have been cited 12 or more times. One of the limitations to this approach is that the h-index is a summary score and will increase as an individual progresses in their career (⁴), making the h-index of limited value when comparing individuals at different career stages. Hirsch also explored the idea of using m, which is equal to the h-index divided by the number of years since an individual’s first published paper. However, because the h-index does not increase linearly over the course of a career, m is not useful either early or late in a career and thus is also not well suited to compare individuals at different career stages. Despite several attempts to improve on the h-index (⁵—⁹), as of yet there is no satisfactory method to control for stage of career, especially for individuals early in their career (⁴).

An alternative, and more common, approach to assess a training program’s success is to focus on the number of scientific publications of its graduates. This approach has the advantage that the number of publications can be divided by the number of years since program completion to compute a "number of publications per year" variable, which may allow for comparison across various stages of careers. One of the limitations to this approach is that it is unclear if all publications should be counted equally. For example, should a first author publication be considered equal to an article in which the graduate is a more distal author? Or should a peer-reviewed article be considered equal to a book chapter or monograph?

The University of Colorado institutional NRSA is in its 29th year of continuous funding. While the institutional NRSA is administratively centered within the department of psychiatry at the University of Colorado Denver, it has always been conceptualized as a multispecialty, multidepartmental, multi-institutional program, drawing faculty mentors from nine departments across four major teaching institutions. Criterion for faculty recruitment has included a primary goal of diversity of methodological approaches; other factors considered include faculty who (a) are senior faculty and have a history of (b) consistent external funding including R01s, (c) consistent peer-reviewed publication record, (d) sufficient resources to allow the postdoctoral fellow to develop his or her own career rather that working exclusively in the mentor’s area of expertise, and (e) some evidence of successful mentoring. As a starting point to address the issue of whether publications should be weighted when measuring trainee outcome, as well as to explore what factors predict publication rates after completion of postdoctoral training, we utilize results from this single long-standing institutional NRSA.

Methods

Participants

The sample consisted of 92 individuals who completed postdoctoral training as part of the University of Colorado’s Institutional NRSA "Development of Maladaptive Behavior" (T32 MH15442) from 1979 to 2004. The sample included 24 medical doctors, 67 Ph.D.s, and one M.D.-Ph.D.; it was 68% female; and, as of 2005, participants had completed the postdoctoral training program an average of 12±7 years earlier (range=1—26 years). Additional demographic information is summarized in Table 1.

Data Collection: Former Postdoctoral Fellows

Current curricula vitae (CVs) were requested from all 92 individuals and received from 89 (97%); publication and funding histories for the remaining three former participants were extrapolated from public databases. Each CV was reviewed and all publications were classified as peer-reviewed or non-peer-reviewed (books, book chapters, monographs, etc.) and, based on the former postdoctoral trainee’s position in the author list, as first author, second author, senior author, or other author. The classification of senior author was used when the former postdoctoral trainee was cited as the last author on a publication of which the first author was identified as a current or former trainee of this individual.

Data Collection: Faculty Mentors

In order to explore which mentor qualities may influence outcome, information was also requested from faculty mentors for this institutional NRSA. By the end of 2005, 35 faculty members had been listed as a potential mentor for at least 2 years of the program (15 with a terminal M.D. degree and 20 with a terminal Ph.D.; 16 women, 19 men); the faculty consists of approximately 20 members at any given time. Gender did not significantly differ on length of time as a mentor, funding history, publication rate, or the impact of their work (h-index). Twenty-nine members of the faculty (83%) have acted as the primary mentor for a postdoctoral trainee. Mentor attrition has occurred for natural career reasons (death, retirement, moving out-of-state); no mentor has been asked to leave the program. Mentors have remained on the postdoctoral faculty for an average of 12±8 years (range=3—27 years). Six of the faculty members are former trainees of this institutional NRSA. An h-index was calculated for each faculty mentor based on a search of citations in June 2007. To match the trainee data, only publications published prior to the end of 2005 were included in calculating the h-index. However, since dates of citation are not easily identifiable, the calculation of the h-index may include situations in which a mentor’s article was cited between January 2006 and June 2007. Thus, the h-indexes may be slightly elevated above what they would have been at the end of 2005. Mentors had an average h-index (³) of 17.8±11.0 (range=5—50); 86% of the mentors had an h-index of 10 or greater. Mentors with the lowest h-index were engaged in fields with historically lower peer-reviewed publishing rates (e.g., nursing and physical therapy). CVs from all 35 faculty members (100%) were examined. Publication records were classified in a manner identical to that described for the former trainees, except that only manuscripts published while the individual was a faculty member were included in number of publications per year. Table 1 summarizes information from the faculty.

Analysis: Defining Outcomes

The number of publications per year is one of the more common measures of an individual’s ability to conceptualize, design, implement, collect data, analyze, interpret, and disseminate research results (¹⁰—¹²). However, there has been little discussion as to whether research output is better determined by counting all publications equally or by weighting publications based on the individual’s contribution (e.g., whether an individual is listed as the first or second author). To address this question, we surveyed the 21 current faculty of this institutional NRSA on the relative weighting that should be applied to each publication; 20 members of the faculty (95%) responded with the results summarized in Table 2. Each publication in each CV was given a weighted score, summed, and then divided by the number of years since completion of the postdoctoral training program to produce a weighted publication score for each individual. The correlation between publications per year and the weighted publication score is a reflection of the variation in publication values across individuals. Publication rates may vary across individuals; however a high correlation between the number of publications and the weighted publication score would suggest that the percentage of publications that are first-author publications would be similar across individuals.

As a secondary outcome measure, CVs were dichotomized according to whether the former postdoctoral fellow had obtained, as a principal investigator, an R01 or similar larger federally funded grant (e.g., an NIH R38 or Veterans Administration Merit Review Grant).

Analysis: Factors That Influence Outcomes

Factors that may influence outcome, including factors related to the trainee (age at entry into the postdoctoral program, gender, race/ethnicity, terminal degree, successful funding of an individual training grant, year of graduation from the program, number of publications before entering the postdoctoral program, and number of publications during the postdoctoral program) and factors related to the mentor (gender, terminal degree, publications per year, h-index, and funding status) were collected for each trainee and their primary mentor. The number of publications published after completion of the postdoctoral program per year was either correlated to the factors (for continuous variables) or compared with a Student’s t test (for categorical variables). All variables that suggested a relationship with number of trainee publications per year (p<0.10) were added into a stepwise multiple regression model.

One common question is whether there are cohort effects on long-term research success, effects that may be due to national issues such as variance in federal funding levels. As an initial examination of this issue, we classified graduates into 5-year graduation blocks (1979—1984, 1985—1989, 1990—1994, 1995—1999, and 2000—2004). Publication rates were then compared across these groups.

Results

As of the end of 2005, the 92 participants had published 1,721 publications; this includes 1,493 peer-reviewed articles (660 of which are first-authored by one of the participants) and 328 other publications such as books, book chapters, or monographs.

Table 2 summarizes the weighting given to each publication classification. Figure 1 displays, for trainees, the correlation between number of publications per year and the weighted publication score (r=0.99, p<0.001). A single outlier averaged 11 publications per year. With that trainee removed, the correlation between number of publications per year and the weighted publication score remained unchanged (r=0.99, p<0.001). The high correlation suggests that the two scores are interchangeable. To be consistent with previous literature, total number of publications is used as the outcome variable for the present analysis.

Table 3 summarizes the impact of individual and mentor variables on trainee publication rate. Additional funded training, such as an individual NRSA or career development award (e.g., NIH-funded K-Award), age at entry into the program, ethnicity, and the number of papers published while enrolled in the T32 program were the only significant predictors of posttraining publication rate; there was a trend for gender to also have an impact. Using a stepwise linear multiple regression with a probability of <0.05 to enter and <0.10 to remove, only additional funded training and number of papers published while enrolled in the T32 program are included in the model (R2=0.22, p<0.001). There was a trend for age at entry into the postdoctoral program to be a significant addition to the model (p=0.06; younger age associated with increased postprogram publication rate).

Twenty percent of graduates have received at least one larger federally funded grant (an NIH R01 or equivalent). Male gender, having received an individual training grant, and more years since graduation from the program are associated with increased likelihood of having received an NIH R01. There was a trend for a younger age at entry into the postdoctoral program and race/ethnicity other than non-Hispanic Caucasian to have an impact. Using a stepwise linear multiple regression with a probability of <0.05 to enter and <0.10 to remove, only additional funded training, male gender, and an increased number of years since graduation from the T32 program are included in the model (R2=0.35, p<0.001).

Using approximately 5-year cohorts, there is no difference between cohorts either in the postprogram publication rates (F=0.62, p=0.65) or in the percentage of graduates who average at least one publication per year (χ2=2.6, p=0.63).

Discussion

The most common outcome measure for training programs is trainee publication rate. However, there has been little discussion of how to weight publications, including the relative value of peer-reviewed versus non-peer-reviewed publications and first author versus other author publications. A survey of current program research faculty provides an initial examination into this issue. Faculty provided weighted values for several different publication types (Table 2) with a peer-reviewed first-author publication standardized to 1.0 and other publications ranging downward to a value of 0.33 for third or more distal author on a non-peer-reviewed publication. On average, weighted publication rates were 76% of the total number of publications, with most individuals having a spread of peer-reviewed and non-peer-reviewed articles with an associated spread of authorship position. Notably, the correlation between the weighted and nonweighted publication rates was extremely high (r=0.99), suggesting that the spread of publication type and authorship position was consistent across individuals. In other words, while the rate of publications may differ across individuals, the percentage of publications that were first author versus other author or peer-reviewed versus non peer-reviewed was similar across former trainees and the two scores can be used almost interchangeably.

The 92 graduates from this postdoctoral program have 1,717 scientific publications with an average of 1.2 publications per graduate per year. Consistent with what is found in family medicine for individuals with at least 1 year of postdoctoral research training (¹⁰), 43% of the former trainees have averaged at least one published paper per year since graduation. However, in contrast to what has been found in family medicine faculty (¹³), a cross-sectional analysis found no correlation between the publication rate and the number of years since graduation. For individuals who received additional research training beyond the institutional NRSA, such as an individual NRSA or a federally funded career development award, the percent of former trainees who average at least one publication per year increases to 85%. Improved outcome in individuals who receive independent research support is consistent with what has been reported in other specialties (¹⁴); however, it is unclear whether the individual training award contributes to a positive outcome or whether successfully competing for an individual training award reflects the same processes which are associated with increased productivity.

In a similar vein, successfully competing for an individual training award is the single greatest predictor of acting as a principal investigator on an NIH-funded R01 or similar grant; 41% of individuals who received an individual training award have later received an R01 compared to 7% of those without a history of individual training awards. The relationship between longer durations since graduation and increased likelihood of successful principal investigator funding is explained mostly by the lack of larger grant principal investigator funding for any graduate less than 6 years removed from the postdoctoral program. This delay in funding is presumably because of the standard recommendation to follow research training with lower budget grant mechanisms prior to attempting a larger grant application.

While the current program provides structured classes in grant writing and ethics, the program is primarily based on an apprenticeship model. Postdoctoral fellows with a Ph.D. did not significantly differ from those with an M.D. degree in either publication rate, in the percentage of graduates who authored one or more publications per year, or in the likelihood of eventually being the principal investigator on a successful larger federal grant. In other words, individuals who came to the program with a history of participation in a degree-granting research-training program (Ph.D. or M.D.-Ph.D.) had no greater success than those who did not. This does not directly answer the question of whether degree-granting pathways should be a component of physician research training, but suggests that they may be of less importance. Interestingly, the number of publications prior to entering the postdoctoral program also had no correlation with posttraining program productivity. Earlier research exposure, as might occur in undergraduate or medical school programs, may influence who enters a research career, but appears to have less effect on the long-term success of that career.

There was no significant effect for any mentor variables—including funding, publication rate, and impact score (h-index)—on trainee publication rate or success at becoming a principal investigator on a larger federally funded grant. No mentor had universally good or bad outcomes; for the eight mentors with five or more trainees over the life of the program, success rates (defined as the percentage of past trainees averaging one or more publications per year) ranged from 14%—63%. Steiner and colleagues (¹¹) have suggested that a mentor with a "national or international reputation" increases the likelihood of trainee success. The requirements for inclusion as faculty mentors in this postdoctoral program include a focus on senior faculty who have already demonstrated consistently high peer-reviewed publication rate and a history of external funding; thus, all of the faculty mentors in this program would meet the criteria of a national or international reputation. There may have been insufficient variability in the current sample to judge the effects of many mentor variables such as publication or funding histories.

A limitation of this analysis is the focus on outcomes of trainees from a single post-doctoral program. Several program variables have remained consistent over the duration of the program, including the administrative structure, the training model, and the focus of the program. Throughout this period the program has had only a single program director (MR) leading a multi-specialty, multi-departmental, and multi-institutional faculty focused on the use of developmental psychobiology to study developmental psychopathology. The program has maintained its independence from departmental administration minimizing outside pressure to promote non-research academic development or fill service needs. The program has consistently utilized an apprenticeship model, an ombudsman, included both physician and non-physician trainees and mentors, and has centered its common training around a twice monthly seminar where more junior researchers, including the post-doctoral fellows, present their research questions and planned methods and analyses for feedback from senior faculty. While holding these variables constant improves the ability to assess individual differences, a limitation of this approach is an inability to assess the impact of these program variables on trainee outcome. Future work comparing this program to similar programs is necessary to address program variables.

Conclusion

This postdoctoral program has a 43% success rate at producing productive researchers—individuals who published one or more articles per year after completing the postdoctoral program. The ability to obtain independent career development funding and the number of publications during the postdoctoral fellowship correlate with after postdoctoral publication rates; other factors such as gender, trainee terminal degree, and mentor publication rate (at least when it is above a minimal rate) appear to have less effect. Similar examination of additional postdoctoral programs will be necessary to examine the role of program variables on outcome.

View Large | Download Slide(.ppt)
Add to My POL

TABLE 1. Demographic Information of Trainees and Faculty Mentors

View Large | Download Slide(.ppt)
Add to My POL

TABLE 2. Average Relative Weight of Publications by Type and Authorship

View Large | Download Slide(.ppt)
Add to My POL

TABLE 3. Impact of Individual and Mentor Variables on Trainee Lifetime Publication Rates

View Large | Download Slide(.ppt)
Add to My POL

FIGURE 1. Weighted Versus Nonweighted Publication Rates

Each publication by each former trainee was classified based on where in the authorship list the trainee presented (e.g. first author, second author, etc.) and whether the publication was peer-reviewed. Each publication was then weighted based on its classification, ranging from a weight of 1 (for peer-reviewed first-author publications) to 0.33 (for third or more distal authorship and not being peer-reviewed). Weighted publication scores correlate strongly with unweighted publication rates (r=0.99).

Acknowledgments

This project was supported, in part, by NIMH (grant # 5T32 MH15442). The authors declare no other competing interests. This work was supported by USPHS Grant MH15442. We would also like to give a special thank you to our trainees, our faculty, and the support of the Developmental Psychobiology Research Group.

References

National Institutes of Health, Office of Extramural Research: National Research Service Award Institutional Research Training Grants, Fiscal Year 2005. Available at http://grants.nih.gov/training/nrsa.htm

National Institutes of Health, Office of Extramural Research: National Research Service Award Individual Fellowships, Fiscal Year 2005. Available at http://grants.nih.gov/training/nrsa.htm

Hirsch JE: An index to quantify an individual’s scientific research output. Proc Natl Acad Sci 2005; 102:16,569—16,572

Burrell QL: On the h-index, the size of the Hirsch core and Jin’s A-index. J Informetrics 2007; 1:170—177

Egghe L: Theory and practice of the g-index. Scientometrics 2006; 69:131—151

Jin BH, Liang LM, Rousseau R, et al: The R- and AR- indices: complementing the h-index. Chinese Sci Bull 2007; 52:855—863

Schreiber M: Self-citation corrections for the Hirsch index. Europhys Lett 2007; 78:30002—30007

Batista PD, Campiteli MG, Kinouchi O, et al: Is it possible to compare researchers with different scientific instruments? Scientometrics 2006; 68:179—189

Sidiropoulos A, Katsaros D, Manolopoulos Y: Generalized Hirsch h-index for disclosing latent facts in citation networks. Scientometrics 2007; 13:253—280

Steiner JF, Lanphear BP, Curtis P, et al: Indicators of early research productivity among primary care fellows. J Gen Intern Med 2002; 17:854—860

Steiner JF, Curtis P, Lanphear BP, et al: Assessing the role of influential mentors in the research development of primary care fellows. Acad Med 2004; 79:865—872

Gill S, Levin A, Djurdjev O, et al: Obstacles to residents’ conducting research and predictors of publication. Acad Med 2001; 76:477

Ferrer RL, Katerndahl DA: Predictors of short-term and long-term scholarly activity by academic faculty: a departmental case study. Fam Med 2002; 34:455—461

Gitlin SD, Yuan Z, Little RJ, et al: Factors that influence successful training and faculty career development in hematology/oncology patient-oriented clinical research. J Cancer Educ 2005; 20:72—78

Download all saved figures as a .zip file.

View Large
Download Slide(.ppt)

Add to My POL