Some random thoughts to wrap up this series.
As before, disclaimers here.
1) The review process-
The NSF review process is very different to that of the NIH. I gather the different Directorates, and even organizations within Directorates, have some leeway in deciding how the review process works. What I'm about to describe is the process my own proposals go through at the BIO Directorate. Your mileage may vary.
Review is basically a two-stage process. First your proposal is sent out to a bunch (at least two, sometimes up to six or seven) ad hoc reviewers, plus at least two members of the review panel. These reviewers send in their reviews via Fastlane. The second step is the review panel. The panel considers all of the reviews (as many as eight or even nine), discusses the proposal and assigns a rank (not a score). The Program Director then takes the rankings and figures out what she can fund.
So you get a bunch of reviews. This is good and bad. On the plus side, you can survive a mediocre review if all the others are stellar. You also get a lot of (hopefully) useful feedback. On the minus side, if one of the panel members who reviewed your grant really didn't like it, your odds of being funded are not great even if all the ad hoc reviewers loved your proposal. As with NIH study sections, you need an advocate on the review panel.
2) Proposal rankings-
As I noted above, reviewers don't score NSF proposals per se. They give them a ranking. The reviewers assign a rank of Excellent, Very Good, Good, Fair or Poor (from memory). The review panel takes those and assigns an aggregate rank of Outstanding, Highly Meritorious, Meritorious, or Non-Competitive. The last two categories won't be funded. Sadly, it is often the case that not all of the proposals ranked as Outstanding can be funded...
WARNING! WARNING! ANECDOTAL STUFF FOLLOWS!
Okay, the following stuff is purely anecdotal - you've been warned. I'm posting it because I've heard the same things from multiple people (n > 6), leading me to think there might just be a grain of truth in each of these...
3) NSF doesn't want young people to fail-
NSF Program Directors have a lot of say over who gets funded and who doesn't. Clearly they can't over-ride the reviewers to much or too many times otherwise they would have a very hard time finding people willing to review proposals. But they can nudge proposals over the funding line if they feel it's warranted. What I have heard from multiple people is that Program Directors will sometimes do this in order to fund an otherwise unfunded young investigator on the verge of their tenure decision. Of course said investigator needs to have been productive enough to warrant this.
This is not something I would bet any money on. Or my tenure. It's obviously best to not to have to rely upon the largesse of a Program Director.
4) Once you're in you're golden-
By that I mean once you're funded by the NSF, as long as you're productive, your Program Director will try to ensure you maintain the ability to be productive. I am by no means suggesting that the Program Director will fund your proposals over higher scoring proposals. Rather that if you're on the funding edge and it's a choice between you as a previously NSF-funded, productive PI and someone else, you will get the nod (remember - this is anecdotal stuff).
I have brought this up hesitantly because it sounds like a mechanism for maintaining the status quo. I would like to think that's not the intent, rather that it's more a matter of Program Directors being loyal to their productive charges. Can it have the side effect of maintaining the status quo? Yes. The someone else is likely a new investigator... Please note that a) THIS IS ANECDOTAL, b) this is somewhat at odds with the previous anecdotal point in this post, and c) I'm not defending this.