I wrote up this little squib on how to review a grant proposal for students in one of our classes. No French here, sorry–our usual exploration of the implications of the statistical properties of language for second-language learning will continue tomorrow.
Global issues in grant reviewing
One of the biggest effects that you can have on science and on the future of science is your work as a grant reviewer for the National Institutes of Health. Your activities in this area will affect not just what kinds of research get funded, but the kinds of approaches that people take to doing that research. To give one big example of the past couple of decades, it’s reasonable to say that some of the genomics-based research that is so fruitful today has been possible only because grant reviewers stopped classifying it all as “just a fishing expedition,” having seen that “fishing expeditions” can be as useful as more traditional hypothesis-driven biology. So, when you get the call (well, today, when you get the email) inviting you to join a “study section,” don’t see it as yet another burden–realize that this is one of your opportunities to have a real effect on science in this country.
What I’ll describe here is my approach to writing a grant review. I’m sure that there are lots of others. Objectively, I can say that (a) this is a more efficient approach than other ways that I’ve tried, and (b) I’ve gotten very positive feedback from multiple scientific review officers at NIH since I started doing it this way, so I have at least one data point that’s consistent with the idea that this is a reasonable approach to the task.
One of the hazards of grant-reviewing is that you may often be asked to evaluate proposals that are outside of your strongest area of expertise. If you really and truly don’t think that you can give something a competent evaluation that’s fair to the investigators, you should certainly tell the scientific review officer so. (Do NOT wait until three days before the reviews are due to do this! You need to take a quick look at all of the proposals that you’ve been assigned as soon as you have access to them, and this should be one of the things that you’re checking for–ensuring that the proposal is not so far out of your area that you couldn’t possibly give it a professionally acceptable review. The other thing to look for: double-check for conflicts of interest.) One of the things that I like about the approach to the task that I describe here is that it can help you in writing reviews of proposals in areas in which you’re not necessarily strong, as the criteria are pretty general to scientific computing research as a field. Science is science, and if the investigators make their points well, you’re going to get them. If they don’t make their points well, then that’s not your fault–point out that they’re not making a very strong case for whatever it is that they want you to accept, and you’ll have helped them to improve their proposal.
Practical approach: reading the proposal
You’re going to have to read the entire proposal thoroughly once, and then you’re almost certainly going to have to skim it at least once. One of your goals is to do that reading/skimming, and the subsequent writing, as efficiently as possible. To that end, you’re going to do two things while you’re reading the proposal. One is that you’re going to highlight things–I’ll tell you what you’re going to highlight in a minute. The other is that you’re going to write very short notes in the margins. So: grab your pen and a highlighter. (You’ll note here that I’m assuming that you’ve printed out your proposals. Yes, I am so old that I still print out proposals.)
First step: read the RFA. You will occasionally see a proposal that just is not “responsive,” as we say, to what an RFA is looking for. This won’t typically be an issue with the “open calls,” but it’s not unusual with others. People have lots of approaches to deciding where to submit their proposals, and for some people, that boils down to submitting the same basic proposal to lots of different places with minor tweaks that try to make them seem relevant to the RFA when in fact, they really just aren’t. Also, some RFAs will have specific requirements that others don’t–maybe a dissemination plan, or a requirement to recruit students from specific under-represented groups, or what have you–and your reviewing responsibilities include ensuring that those specific points are addressed.
Having read the RFA, you’re now ready to start reading the proposal. When you do that, you’ll want to read it in such a way as to make it easier to write the review in the format in which it’s required to be written. At the moment, here’s what the required format is:
- An “overall impression/summary.” This section describes what the proposal is about. Additional details summarize the strong and weak points very broadly. Overall, the picture that you paint in the impression/summary should make sense in terms of the overall numerical score that you give, and vice versa.
- Specific scores for innovation, etc.
- For each of those areas, specific lists of strong points and weak points.
So: you’re reading with your highlighter and your pen in hand. You’re going to highlight material that gives a good summary of what the proposal is about. You are completely free to use this material in your overall impression/summary write-up, and there’s some advantage to doing so, as it minimizes the chances of you mis-quoting the proposal. This might come from the introduction to the proposal, or it might be scattered throughout it, depending on the skill of the investigators who wrote it.
As you read, you’re going to mark in the margins anything that you think constitutes a strong point or a weak point, and which area of the review (innovation, approach, etc.) you think it’s a strong point for. You might also scribble a couple words in the margin to remind yourself what it was, exactly, that struck you as a strong or as a weak point. For example: the proposal says that Despite the obvious potential for accelerating biomedical research with better recognition and normalization of cell line names in GOA records, this problem has not previously been studied. In the margin, maybe you write + Inn. In other words: here’s something to list under “strengths” in the Innovation section of the review. Maybe the proposal says that We will write a look-up dictionary for every possible syntactic structure of the English language. Now you write – App in the margin, since this is something that you’ll want to mention under “weaknesses” in the Approach section–there is an infinite number of English sentences, so this approach is not very likely to work.
- Talk about the work, not about the writer. Not the PIs do not make a convincing case that… but the proposal does not make a convincing case that… (The one exception to this is when you’re specifically asked to evaluate the investigator–more on that below.)
- Try to think about the reviewing process in terms of being helpful, rather than fault-finding, and at least write as if that’s the case. So, for example, write The proposal could be improved by… rather than the proposal does not… Bear in mind here that you don’t just want to sound like you’re being helpful–you should actually try to be helpful! You don’t have to rewrite the proposal for them, but do approach this task from the good place in your heart.
- Always try to find a strong point for every section of the review. They can be extremely generic–a strong point of the proposal is that cancer is a serious problem. They can be extremely specific. For example, if reviewing a spectacularly bad proposal on predicting sub-cellular localization from amino acid composition from an investigator at the University of Lower Slobovia, you might say something like this: The University of Lower Slovobia is very strong in the field of marmot communication. The scientific review officer will realize just fine that marmot communication expertise is not relevant to the likelihood of success of a proposed project on predicting sub-cellular localization from amino acid composition. If you say in the Significance section that Cancer is a fatal disease, the scientific review officer knows that if your only strong point is that cancer is a fatal disease, then you absolutely weren’t able to find any strong point of importance. And, don’t worry that saying one good thing about a bad piece of work will lead to crappy science getting funded–the fact that you’ve got one strong point and eight weak points will make it clear that the proposal needs a lot of improvement. You should, however, worry that not saying even one good thing about a bad piece of work might give a troublesome investigator an opening for hassling the scientific review officer about objectivity, or even just be unnecessarily hurtful to the investigator. You can always find one strong point, or almost always:
- Innovation: Curing cancer would be novel and important.
- Approach: Java does a good job with memory management, so this is a good choice of programming language for the project.
- Distribution plan: SourceForge is a relatively stable platform for distributing code.
- …you get the idea. These might sound flip and sarcastic to you, and it could be accurate to describe them that way, but if read on the surface, they’re pretty anodyne. I have occasionally run this specific kind of thing by scientific review officers to make sure that they don’t come across as sarcasm, and I have never gotten negative feedback on them. (That’s not to say that I won’t during the next review cycle–but, so far, so good.)
Those are general approaches to writing the review as a whole. There are also some specific things to think about when reviewing specific sections of an informatics proposal. For the Innovation section: is there innovation both in terms of the intended application (finding drug side effects that might indicate potential new uses in electronic health records, mining casual mechanisms from scientific literature, whatever) and also in terms of the informatics aspect of the work? This is potentially one of the toughest areas to evaluate if you’re not an expert in the field, but ultimately, it’s up to the investigators to make a strong case for the innovation of the project. ‘Fess up to gaps in your expertise–let the scientific review officer decide whether or not it’s OK for you to review the proposal.
Regarding the approach: here you’re looking for things that affect the likelihood of success of the work. This is potentially another tough area in terms of dealing with not being an expert, but there’s still a lot that you can do.
- Look for aspects of the approach that clearly require specialized expertise in development and in testing. If the proposal has budgeted for a doctoral student to program a system requiring extensive experience with database design and security issues–something that you might conceivably hire a $200-an-hour consultant for–that’s a very valid thing to point out as a weakness of the proposal, as it just isn’t very likely to lead to success.
- Who defines the use cases for the software? You want to see someone who is a potential user doing this, not, say, a software developer, unless the intended users are software developers.
- Does the proposal include explicit plans for software testing, for ensuring robustness of the applications, and for maximizing the possibility of repeatability and reproducibility of the work? This is a rapidly growing area of concern in biomedical science, and it is likely to become much more so in the immediate future.
- Don’t forget about data. Larry Hunter’s definition of bioinformatics is “doing original science with other peoples’ data.” Doing this requires that the data be available to you. If data is required: have the investigators demonstrated that data is available? By this time in your career, you’ve probably discovered that difficulties with getting access to data is one of the most common causes of failed course projects, missed deadlines, and the like.
- Reviewers usually want to see back-up plans in case the proposed approach doesn’t pan out. Personally, I usually write these as things that are likely to be successful based on previous work, but that aren’t as innovative as what I’m actually proposing to do. You see proposals getting critiqued for not having back-up plans, but reviewers don’t typically find fault with the back-up plans themselves. (On that last point, your mileage may vary. I once wrote a proposal with a co-worker. We had very explicit back-up plans. One of the reviewers wrote: The back-up plans seem like a good idea. Why don’t the investigators just do that? Oh, well.)
There is a specific section of the review template for the investigator(s). This is the one place where you can’t avoid talking about the people involved (as opposed to my advice above to make your review about the work/the proposal, not about the investigators). This is an area where you can be really hurtful, which is not kind. It’s also a place where you can cause problems for the scientific review officer by saying things that a disgruntled investigator might be able to point to as evidence of lack of objectivity. You’re safe if you stick with the kinds of things that people usually address in this section:
- Is this a senior person (in which case you need to check that they’ve actually committed time to the work)?
- Is this a junior person (in which case you might want to point out specific evidence that suggests that they could manage the project, if you want to support the proposal, because lack of such evidence is a common way of trashing a proposal)?
- Has the person worked in this area before, or are they trying to strike out in a new direction? The latter can be fine, if they have collaborators with suitable expertise.
- Has the PI worked with the other members of the team before? If so: strong point. If not: weak point.
- At some point, you need to look at the entire group of people involved. You might do this in the Investigator section, or in the Approach section. You’ll at least need to look at the types of people involved. (For an example of the latter: the PI might say that they would hire, say, a statistician, without specifying who it would be.) An informatics project that’s likely to be successful is going to have the right mix of researchers and technical people. At the risk of being redundant: a frequent problem that you’ll see is a researcher being tasked with implementing a super-complex computational platform that requires professional software development and testing experience more so than research abilities. Look for plausibility of software development goals given the number of people on the budget–an enormous project is not going to be built with just one software developer, but that may be all that you’ll find on the budget. Look for adequate software testing; look for requirements for special expertise, both in developing requirements, in developing software, and in testing software.
- Another point on the entire group of people: if there is already evidence that they can work productively together, you should count that as a strength. Look for papers published together, previous grants involving the same group, etc. Conversely, if you are reviewing a big, complicated proposal with lots of dependencies between groups and potentials for lack of communication and the like, and the people have no history of ever having worked together before, then you should consider pointing that out.
- UI design issues can be a big issue. Many people say that they plan to build a user interface, but don’t seem to think about how they will design the interface, or how they will evaluate it. In particular, people often don’t seem to plan ahead to involve users in the interface design and evaluation process. Having developers design user interfaces typically doesn’t work out well, and not involving users in this is a very legitimate weakness.
- What counts as success, and how will you know if it’s been achieved? You don’t have to agree with the way that the investigators think that this question should be answered, but the investigators should at least tell you how they think it should be answered. Beware of statements like we will build a system that predicts protein subcellular localization that aren’t followed by some implicit definition of what build and predict mean. How will the program officer know whether or not the protein-subcellular-localization system has, in fact, been built? When the investigators go back for a renewal of the grant, how will the reviewers of the new proposal know whether or not this is something that was accomplished? How will the investigators themselves even know when they’re done? If I write a Python script with a predictProteinSubcellularLocalization() function that works for exactly one hard-coded protein, have I accomplished the aim? What if I build a beautifully engineered system with a user interface so intuitive that 5th-graders are now predicting subcellular localization just for fun from the comfort of their living rooms–but the predictions are never, ever right? What if I build a localization system, but it can only predict a subcellular localization once before the code auto-deletes? What if I build a system that has 2,000,000 users three weeks after I release it and quickly leads to the discovery of cures for every existing type of cancer, but I never publish a paper about it? Does that count? The investigators don’t have to define success the same way that you would, but they should be specific about what they’re going to produce.
The discussion session
There’s a rhythm to discussions of grants. The typical NIH routine is that:
- Reviewer 1 presents a synopsis of the proposal, including the goal, the approach, and the main strong and weak points.
- Reviewer 2 does not present a synopsis, and typically doesn’t repeat Reviewer 1’s points, although they may begin by saying something like I agree with Reviewer 1’s description of the proposal. Reviewer 2 then usually adds any additional things that they think need to be pointed out.
- Reviewer 3 writes up only a short review, and doesn’t typically say much during the discussion, beyond adding any additional things that they think weren’t addressed by Reviewers 1 and 2.
When you’re Reviewer 1, be true to your analysis, but be as humble as you always are (or should be), and give as much real consideration to other peoples’ analyses as you always do (or should). (See this post for information about how to use a lack of humility and the failure to consider that other people might have valid analyses, too, to fully exercise your unemployment benefits.)
Learn from the experience
As with as anything else, you want to learn from the experience of reviewing grants. What you’re going to learn: how to write better grant proposals yourself. Thinking seriously about the critiques that you get on your own proposals is an excellent way to improve them–paying close attention to the issues that an entire room full of reviewers raise about proposals on a wide variety of topics is even more efficient.
Zipf's Law 