@foaylward

In terms of the "competitive not discussed" problem --- yes, 💯. I have been talking to a lot of colleagues about this (because my grants also are ending up in this unpleasant space) and everyone agrees that these are the grants that need the most discussion. Some of these reasons include

- these are the grants that are addressing complex special issues that people don't understand and would benefit from the broader expertise on the panel.

- these are the grants where a reviewer made a mistake (such as not understanding the purpose of an R21 😠!)

- these are the grants that have disagreements between reviewers, where one reviewer liked them and one reviewer didn't. Most of these grants are not in this 25%-50% space because all three reviewers think they are a 33% Rather they are often because some reviewers liked it a lot, and a some hated it.*

* Historically, disagreements like this would lead someone to "rescue" the grant for discussion because they wanted a fair review to address the discrepancy. However, over the last decade or so, #NIH has moved from grant review as "provide feedback on the science" to grant review as "order quality for funding decisions", and study section chairs have been asking people "Will this move the grant enough to matter? If not, then do we need to spend the time on it?" Since funding is now at something insane like 5%, people are reluctant to ask study section to spend time discussing a grant to move it from 33% to 20%.

This is also coming from colleagues on study section who observe that study section spends a lot time discussing grants everyone agrees on (particularly at the good-score end).

#grants #grantReviews

2/2

#NIH #grantReviews

I think this is a great idea. Here's what I wrote:

This is an excellent suggestion and a fantastic idea. I am strongly in support of it. I have been dealing with grants and grant reviews for over 25 years. In my extensive experience, reviews are not accurate beyond about +/- 20%. (I suspect it’s larger than that, but I feel 20% is a safe conservative estimate.)

This new process will allow program to make better decisions taking larger scale information into account, such as ensuring workforce continuation (considering whether funding prevents a lab’s collapse and the loss of valuable process knowledge), ensuring a broad portfolio of research structures (risky new ventures, solid blue-chip stocks), and programmatic goals.

This proposal may also help reduce the burden on study section. Under this new system study section really only needs to discuss (a) grants at a boundary between categories and (b) grants that someone feels are miscategorized.

I would also like it if program is not able to see the scores.

I would also like that the reporting should not include partial scores (like “environment” or “impact”) as those reveal scores that can be used. I would like it to be based on the actual reviews identifying specific strengths, weaknesses, and concerns that program can check with the applicant about.

PS. The reason for the noise in the scores is due to a variety of features, which are probably not solvable. Much of the noise comes from other problematic structures of study section: Because NIH grants are reviewed in depth by only three people, you have tremendous shot noise in the data. Because NIH grants are reviewed by broad field experts and not specific field experts, reviews are often mistaken in details that require additional information from the applicant, but grant review does not provide for responses. (NIH has been slowly destroying the “reply/respond” cycle that used to be present 25 years ago, first limiting to A1/A2, then limiting to A1, and now specifically telling reviewers not to make their decision based on the reply to reviews in the A1.) Finally, because the stupid non-linear 1-9 scoring system (much worse than the 1.0-5.0 scoring system, which was linear) is misunderstood by many reviewers, different reviewers center their scores at different points, adding to the noise. However, in the end, science is about exploration. Until we actually go look, we won’t know what we will find. So unless we are going to fund retrospectively (you did great work for the last 5 years, here’s another 5 years), we are probably stuck with this kind of noise. The process should respect that noise and not try to pretend precision that we simply do not have in grant review.