Why does the NIH have to tell us how to conduct research?

by Anita Bandrowski

Dr. Landis, and the rest of the 'who-is-who-in-science' have recently published a paper (PMID:23060188) in Nature about the reporting practices in biological sciences. This follows some of the reports that have ruffled the scientific community recently, such as the study by Amgen scientists (PMID:22460880), which reported that less then 10% of major cancer studies were replicable. Now, however you may feel about Amgen and the rest of the biotech industry, what is the point of publishing research that can't be replicated? Are we, as a community, falling down on the job? We are paid to do research, train students in the scientific method and publish that research. We are the wise old men (and women) who should hold the line against the fast, the easy and the popular. Instead, we as a community are falling into various pitfalls that we ought not to fall into. Indeed, one satirical statistician pointed out (repeatedly, but apparently Amgen and others have yet to listen) that just about every report in one of the top tier journals is an anomaly (PMC1182327). We could perhaps pause, and think about this.

The issues that were brought up by Dr. Landis sound like what my advisor was trying for years to beat into my head many many years ago when I was a young and naive graduate student.

1. Report how you randomized the study.

2. Blind study design is better than not blind.

3. Estimate sample size by using an actual formula (its easy just manipulate the t-test formula to solve for n).

4. Tell us (or better still show us) what you have done with data (if you used statistical software, which one, which version, which machines).

Why does a director of the NIH have to tell the scientific community this? Is this no longer taught in graduate school? I am not sure how many other people have had weeks of experimental findings thrown (when data was still on paper) into the trash due to inadequate controls, but seems like perhaps it should be done a lot more often.

I am certainly no expert on teaching or training, but interacting with some great teachers I have learned that "students will always hate advisors, its just a matter of when" (Glenn Hatton, ~1999). In my case, this happened when my paper was returned marked up stating something to the effect of: "provide softer toilet paper next time." Although I hated my advisor at the time, I really wish I had kept that gem.

Perhaps as a community, we could rethink the incentive structure in biology because papers that are reasoned, well controlled, thorough and meticulous are seldom published in very high impact journals because they are not sufficiently novel or surprising. We hire scientists based on having published in one of the 'novelty seeking' journals. Grants are often penalized because they are not sufficiently novel or exciting. Meaning that, careful reasoned research is being penalized and seeking novelty is rewarded? Would Darwin have gotten funding from NIH or would it not be sufficiently novel to measure beak size?

Is this the way that science ought to be incentivized?