Don't you have anything better to do?
Tuesday, January 27, 2009
Global time wasters
My, my there are a lot of you, and from all over!
Figure 1: Map denoting where the visitors to this blog hail from (red dots).
Don't you have anything better to do?
Don't you have anything better to do?
Thursday, January 22, 2009
How Many Papers for Tenure?
The title of this entry comes from a Google search a recent visitor to my blog used.
Good question.
I can't answer it.
The only people who can give you some kind of answer to this are senior faculty in your department (and possibly college), and yourself.
But you absolutely must know the answer very early in the tenure-track process. Preferably before you officially start your position. At the very latest six months in.
So how do you find out how many papers you need? You need to do two things:
1) Ask. Ask early and often. Ask the senior faculty in your department. If you've met some, ask senior faculty in related departments within your college (e.g. within medicine, or within arts and sciences). You may not get a straight answer. You may get several different answers. Hopefully you will get some kind of answer.
2) Look it up. Find out how many publications the last few people who made tenure in your department had. No one tenured in the last few years? Look up the publication totals of recently tenured people in related departments within your college.
Once you've done both of the above, take the largest of the answers you've found (they may not match up).
That's the absolute minimum number of publications you are aiming for.
Absolute minimum. You want more. You may need more.
The answer to the question "how many papers needed for tenure?" is not set in stone.
[UPDATE] Do apply some common sense to the above. Let's say your senior colleagues provide you with a consensus estimate of X publications, and the last three people to get tenure in your department had (X+1), (X+2), and (X+10) publications. Take (X+2) as your minimum goal.
Good question.
I can't answer it.
The only people who can give you some kind of answer to this are senior faculty in your department (and possibly college), and yourself.
But you absolutely must know the answer very early in the tenure-track process. Preferably before you officially start your position. At the very latest six months in.
So how do you find out how many papers you need? You need to do two things:
1) Ask. Ask early and often. Ask the senior faculty in your department. If you've met some, ask senior faculty in related departments within your college (e.g. within medicine, or within arts and sciences). You may not get a straight answer. You may get several different answers. Hopefully you will get some kind of answer.
2) Look it up. Find out how many publications the last few people who made tenure in your department had. No one tenured in the last few years? Look up the publication totals of recently tenured people in related departments within your college.
Once you've done both of the above, take the largest of the answers you've found (they may not match up).
That's the absolute minimum number of publications you are aiming for.
Absolute minimum. You want more. You may need more.
The answer to the question "how many papers needed for tenure?" is not set in stone.
[UPDATE] Do apply some common sense to the above. Let's say your senior colleagues provide you with a consensus estimate of X publications, and the last three people to get tenure in your department had (X+1), (X+2), and (X+10) publications. Take (X+2) as your minimum goal.
Tuesday, January 20, 2009
Dear Future Applicant for Tenure and Promotion
Your time on the tenure track is thundering its way towards the big decision. Soon you are going to have to put together that thick wad of documents known as your tenure packet/dossier/file. All of the documents in that packet are important, but some are more important than others. None are more important than your curriculum vitae. This document, your c.v., is a summary of almost everything needed to evaluate you for tenure and promotion. The other documents just provide the details.
So, dear applicant, please take the time and spend the effort to make your c.v. a document that will help those evaluating you come to the right decision. Because we really do want to see you be promoted with tenure. (Unless of course one of us is an obnoxious dickwad.)
You can ease our task by making all the information we need easy to find and read. Put the important stuff up front, and leave out the truly unnecessary.
And please don't use your c.v. to demonstrate your artistic nature. Please. Don't.
Do have a neat, logically organized c.v. And spell check it. A sloppy, disorganized c.v. makes you look... well, sloppy and disorganized. Not really what I want in a colleague.
Here are some further thoughts on some (not all) of the basics:
Background/Training/Education
My personal preference is for this to be in reverse chronological order, going from your current position back to your undergraduate schooling. Leave out where you went to high school* - really, we don't give a rat's rear end. And those perfect SAT/GPA/GRE scores you're so proud of?* Not of any interest to us when it comes to tenure decision time.
Publications
Reverse chronological order please. That Journal of Biological Chemistry paper you published while in elementary school? Very impressive, but not going to count towards tenure. Put the papers you've generated as an independent PI first. Those are the only ones that count. And do us a favor. Make it very clear what papers came out of which positions - I personally like subheadings denoting "Papers arising from my postdoctoral work", "Papers arising from my graduate work" etc. It's important to include these because they establish your history. But they won't count towards tenure.
And those manuscripts on your postdoctoral work you published while on tenure track? Sorry, they don't count towards tenure. They belong in the "Papers arising from my postdoctoral work" group. No, really, they don't count. Ever.**
What about papers "in preparation"? Not going to count. Don't bother with them.
Okay, if you really, really must. One "in preparation". Two at the very most. They still won't count. Three or four "in preparation"? That's asking for trouble. Unless you've published a gazillion papers while on tenure track - but they still won't count. If you're a little short on published papers and list three or four "in preparation"?* You've just given me the impression you have trouble finishing things, even with the tenure guillotine hanging over your head.
Ten "in preparation"?* You're a delusional fool. Granted, I've met some tenured delusional fools, but to a person they hid their delusional foolishness until after they got tenure.
Funding
Start with current/active funding. That's the most important by far. Actually it's really all that counts. Then list pending. If you have a revised grant pending that got a good score the last time it was reviewed, give us the score. Expired/previous funding comes last. Within each subgroup, reverse chronological order please.
Awards
You won a trophy for badminton at high school? How nice. Doesn't count. Please just list the relevant stuff. Preferably in reverse chronological order.
Professional Society Membership
Listing this doesn't count for much, but not listing it looks... odd.* It suggests that you're not involved in your field outside of your own research program.
Internal Service
Please indicate what level each committee is. A Graduate Curriculum Committee at the University level is a very different beast to one at the Department level. President of your undergraduate chemistry club? Doesn't count.
External Service
You do have some, right? List the journals you reviewed manuscripts for and the agencies you reviewed proposals for. Editorial boards. Professional society activities. These are important. To get tenure and promotion you need to demonstrate that you have national standing. Recent national standing.
Bottom line: Put the stuff that counts up front. You do know what counts at your institution, right?
* Yes, I have seen this in a c.v. within a tenure packet.
** This topic deserves a post all of it's own.
So, dear applicant, please take the time and spend the effort to make your c.v. a document that will help those evaluating you come to the right decision. Because we really do want to see you be promoted with tenure. (Unless of course one of us is an obnoxious dickwad.)
You can ease our task by making all the information we need easy to find and read. Put the important stuff up front, and leave out the truly unnecessary.
And please don't use your c.v. to demonstrate your artistic nature. Please. Don't.
Do have a neat, logically organized c.v. And spell check it. A sloppy, disorganized c.v. makes you look... well, sloppy and disorganized. Not really what I want in a colleague.
Here are some further thoughts on some (not all) of the basics:
Background/Training/Education
My personal preference is for this to be in reverse chronological order, going from your current position back to your undergraduate schooling. Leave out where you went to high school* - really, we don't give a rat's rear end. And those perfect SAT/GPA/GRE scores you're so proud of?* Not of any interest to us when it comes to tenure decision time.
Publications
Reverse chronological order please. That Journal of Biological Chemistry paper you published while in elementary school? Very impressive, but not going to count towards tenure. Put the papers you've generated as an independent PI first. Those are the only ones that count. And do us a favor. Make it very clear what papers came out of which positions - I personally like subheadings denoting "Papers arising from my postdoctoral work", "Papers arising from my graduate work" etc. It's important to include these because they establish your history. But they won't count towards tenure.
And those manuscripts on your postdoctoral work you published while on tenure track? Sorry, they don't count towards tenure. They belong in the "Papers arising from my postdoctoral work" group. No, really, they don't count. Ever.**
What about papers "in preparation"? Not going to count. Don't bother with them.
Okay, if you really, really must. One "in preparation". Two at the very most. They still won't count. Three or four "in preparation"? That's asking for trouble. Unless you've published a gazillion papers while on tenure track - but they still won't count. If you're a little short on published papers and list three or four "in preparation"?* You've just given me the impression you have trouble finishing things, even with the tenure guillotine hanging over your head.
Ten "in preparation"?* You're a delusional fool. Granted, I've met some tenured delusional fools, but to a person they hid their delusional foolishness until after they got tenure.
Funding
Start with current/active funding. That's the most important by far. Actually it's really all that counts. Then list pending. If you have a revised grant pending that got a good score the last time it was reviewed, give us the score. Expired/previous funding comes last. Within each subgroup, reverse chronological order please.
Awards
You won a trophy for badminton at high school? How nice. Doesn't count. Please just list the relevant stuff. Preferably in reverse chronological order.
Professional Society Membership
Listing this doesn't count for much, but not listing it looks... odd.* It suggests that you're not involved in your field outside of your own research program.
Internal Service
Please indicate what level each committee is. A Graduate Curriculum Committee at the University level is a very different beast to one at the Department level. President of your undergraduate chemistry club? Doesn't count.
External Service
You do have some, right? List the journals you reviewed manuscripts for and the agencies you reviewed proposals for. Editorial boards. Professional society activities. These are important. To get tenure and promotion you need to demonstrate that you have national standing. Recent national standing.
Bottom line: Put the stuff that counts up front. You do know what counts at your institution, right?
* Yes, I have seen this in a c.v. within a tenure packet.
** This topic deserves a post all of it's own.
Monday, January 12, 2009
Writer's block
Anyone know a good cure?
I'm trying to write what should be the last paper from my old, almost dead research program, but just can't raise the enthusiasm to get it done. I need to get this off my desk for a bunch of reasons, not the least of which being that one of my former undergrads worked her butt off collecting the data for this paper. She deserves the first authorship. She is now off at med school, so the writing is up to me.
I'd rather be writing about my new research...
I'm trying to write what should be the last paper from my old, almost dead research program, but just can't raise the enthusiasm to get it done. I need to get this off my desk for a bunch of reasons, not the least of which being that one of my former undergrads worked her butt off collecting the data for this paper. She deserves the first authorship. She is now off at med school, so the writing is up to me.
I'd rather be writing about my new research...
Monday, January 05, 2009
It's a New Year!!! Woohoo!!!
Most years in early January I waste time bemoaning the fact that I didn't achieve everything I wanted to in the previous year. Not this year. Did I get everything done I wanted to last year? Hell no. I fell about two manuscripts and one grant submission short of my goals. So what? I managed to rebuild my dying research program and renew my NSF funding. IMHO, a good year.
This year seems somehow more exciting and full of new possibilities. My new research project is up and running (albeit slowly), I'm laying the groundwork for a second, equally exciting new project and I'm restocking my lab with new personnel. Right now, the academic life is good.
What about the funding downturn and failing economy? Not gonna let that stop me. Uh uh. Wouldn't be prudent. Someone's getting funding. Might as well be me. Besides, bitching and moaning about such things is not only a waste of time, but also makes you a less than desirable colleague. I'd rather be working on my kick-ass science.
How about you?
This year seems somehow more exciting and full of new possibilities. My new research project is up and running (albeit slowly), I'm laying the groundwork for a second, equally exciting new project and I'm restocking my lab with new personnel. Right now, the academic life is good.
What about the funding downturn and failing economy? Not gonna let that stop me. Uh uh. Wouldn't be prudent. Someone's getting funding. Might as well be me. Besides, bitching and moaning about such things is not only a waste of time, but also makes you a less than desirable colleague. I'd rather be working on my kick-ass science.
How about you?
Thursday, December 18, 2008
Are You a Rockstar Scientist!!!!!?
Many, many years ago, while I was a grad student, New Scientist magazine published a quiz titled "Leader of the Lab." I have shamelessly stolen borrowed and altered what I remember of that for the following:
Answer the following multiple choice questions to determine if you have what it takes to be a Rockstar Scientist!!!!! Keep a note of your answers and use the grading system at the end of this post. No cheating!
1) You've just started your tenure track faculty position. You're trying to get your lab up and running when your newly hired tech comes into your office clearly upset. His mother has just been diagnosed with a terminal disease and has only a week or so left to live. The tech hasn't as yet earned any vacation or sick leave time, but would like to take time off to spend with his dying mother. What do you do?
2) Two graduate students want to join your lab. Both have indicated an interest in the same project of potential high impact. What do you do?
3) An undergrad from one of your classes approaches you about doing research for credit in your lab. She is very interested in your work and is going to apply to grad school. You've never had an undergrad do research with you before, but are well aware that it's a substantial time and effort commitment on your part. What do you do?
4) You will soon be coming up for tenure. You have plenty of grant money and a bunch of published papers, including two in the C/N/S glamormagz. A senior colleague is planning to go on sabbatical and asks you to teach one of his courses while he's gone. It's very much in your area of expertise and your colleague has offered you his Powerpoint slides and lecture notes. What do you do?
5) You're now an associate professor with tenure. Professor Bigwig, a leader in your field, is coming to give a seminar in your Department. You know that she's organizing next year's SuperBig Meeting in your field. By sheer coincidence you are currently reviewing a renewal of the major grant that Bigwig needs to keep her laboratory running. What do you do?
6) You're sitting on a grant review panel when you are assigned a proposal from a new investigator who is proposing to do precisely what you were going to put into a grant application you're currently writing. What do you do?
7) You are now a full professor. An assistant professor in your Department, who is coming up for tenure, has been collaborating with you. You've each put in the same amount of time, effort and $'s. The manuscript is written and you're deciding on authorship prior to submission (a postdoc is clearly first author). What do you do?
8) You've been very successful at grant writing and now have a lab full of smart postdocs. What do you do?
9) When you first arrived at your University your Department purchased a confocal microscope you needed. The deal was that you would be in charge of this instrument, but that others in the Department would have access and you would provide the training. Over the years you have been doing less and less confocal work, to the point that you now no longer need such an instrument. What do you do?
10) A young, completely unknown, recently graduated PhD from Estonia emails you a rough draft of a manuscript for you comments. Upon reading the manuscript you realize that this is Nobel-prize stuff. What do you do?
Answer Key:
For each "a" give yourself one point, for each "b" two points, each "c" three points, and each "d" four points.
If you scored:
10-20
Sorry, you are just too nice to ever be a Rockstar Scientist!!!!! Your colleagues really like you and you are certainly popular among the grad students. But stardom will forever be out of reach.
21-30
Not going to make it to Rockstar Scientist!!!!! You try, but that pesky conscience keeps getting in the way.
31-39
You're almost a Rockstar Scientist!!!!! But not quite. What's holding you back? Couldn't sell the confocal? Missed the plane to Stockholm?
40
You are truly a Rockstar Scientist!!!!! Congratulations! It must be great not having to write grant proposals anymore. What did you get for the confocal and how was Stockholm?
Are You a Rockstar Scientist!!!!!?
Answer the following multiple choice questions to determine if you have what it takes to be a Rockstar Scientist!!!!! Keep a note of your answers and use the grading system at the end of this post. No cheating!
1) You've just started your tenure track faculty position. You're trying to get your lab up and running when your newly hired tech comes into your office clearly upset. His mother has just been diagnosed with a terminal disease and has only a week or so left to live. The tech hasn't as yet earned any vacation or sick leave time, but would like to take time off to spend with his dying mother. What do you do?
- Of course he should go. He should take as long as is necessary. And he shouldn't worry about being paid - you'll make sure he is.
- He can go. Although you feel bad about it, it does have to be leave without pay.
- He can go when he finishes up all the experiments he was supposed to have done during that time. And it's leave without pay.
- Tell him that's what weekends are for.
2) Two graduate students want to join your lab. Both have indicated an interest in the same project of potential high impact. What do you do?
- Explain the situation to both and try to work out who gets the coveted project. Do your best to come up with a second project of equal potential for the other student.
- Give the best student the project and assign the second student something else.
- Give the best student six months on the coveted project. If after that time they haven't made sufficient progress, give the project to the second student.
- Put them both on the project and tell them whoever gets Nature/Science/Cell-worthy data first wins. The loser gets the boot.
3) An undergrad from one of your classes approaches you about doing research for credit in your lab. She is very interested in your work and is going to apply to grad school. You've never had an undergrad do research with you before, but are well aware that it's a substantial time and effort commitment on your part. What do you do?
- Agree and set up a schedule where you can work with her until she's up to speed in the lab. Thereafter make sure you meet with her at least weekly.
- Agree, then assign her to one of your grad students. Check up on her progress about once a month. Let the grad student decide her grade at the end of the semester.
- Agree, then assign her to your tech. Don't bother checking up on her progress and assign her a C at the end of the semester.
- Agree, then make her the lab dishwasher. Make yourself unavailable to talk with her once she's started. Give her an F for not having produced any data.
4) You will soon be coming up for tenure. You have plenty of grant money and a bunch of published papers, including two in the C/N/S glamormagz. A senior colleague is planning to go on sabbatical and asks you to teach one of his courses while he's gone. It's very much in your area of expertise and your colleague has offered you his Powerpoint slides and lecture notes. What do you do?
- Agree to teach the course. It won't be that much work and will earn some good will.
- Agree to teach the course but insist that your colleague writes a glowing paragraph about your selflessness in his letter supporting your tenure.
- Tell your colleague you already have too much teaching.
- Say no, then go tell your Department Chair that, with tenure coming up, you need to lighten your teaching load. Insist your senior colleague (the one who wants to go on sabbatical) is not only the best replacement, but the only person in the Department capable of giving your lectures.
5) You're now an associate professor with tenure. Professor Bigwig, a leader in your field, is coming to give a seminar in your Department. You know that she's organizing next year's SuperBig Meeting in your field. By sheer coincidence you are currently reviewing a renewal of the major grant that Bigwig needs to keep her laboratory running. What do you do?
- Meet with Bigwig, tell her about the latest findings in your lab, and pay her all the respect she is due.
- Meet with Bigwig and try to dazzle her with your latest findings in the hope she will consider you for a speaking slot in Superbig Meeting.
- Meet with Bigwig and try to charm her into giving you a speaking slot. If that doesn't work, outright demand a speaking slot using your latest C/N/S publication as justification.
- Meet with Bigwig and point out what a wonderful speaker you are and that you noticed there's an empty slot for a plenary speaker. All while tapping on that copy of her proposal you left lying in full view on your desk.
6) You're sitting on a grant review panel when you are assigned a proposal from a new investigator who is proposing to do precisely what you were going to put into a grant application you're currently writing. What do you do?
- Let the program officer know the situation. Review the proposal as objectively as possible. If the new investigator's proposal receives a fundable, or near fundable, score, rethink your proposal.
- Tell no one, but review the proposal somewhat objectively. Give it the score you think it deserves, given that this is a new, unproven investigator... Submit your proposal as planned.
- Review the grant, but, no matter how good it is, give it a score that guarantees it won't be funded. Insist on additional preliminary data that will take at least a year to generate. Submit your proposal as planned.
- Trash the grant, writing a review that sends the message that the proposed work is crap, and that the new investigator has never had and never will have a decent idea in their life. Then cut and paste from their proposal into your proposal.
7) You are now a full professor. An assistant professor in your Department, who is coming up for tenure, has been collaborating with you. You've each put in the same amount of time, effort and $'s. The manuscript is written and you're deciding on authorship prior to submission (a postdoc is clearly first author). What do you do?
- Give the junior person senior authorship - it will be a big help in getting them tenure.
- Insist on being co-corresponding author with your junior collaborator, but let them be last author.
- You're senior. Obviously you'll be senior author.
- Collaborator? What collaborator?
8) You've been very successful at grant writing and now have a lab full of smart postdocs. What do you do?
- Keep doing what you're doing. Provide the best environment you can for the people in your lab, encourage them to write proposals for their own funding, and make sure they move on to bigger and better things when the time is right.
- Keep those postdocs working hard. If they can keep the data flowing and you can get your own proposals funded, they can apply for their own funding.
- Have your postdocs write grant proposals for you. Apply your editorial magic to them before putting your name on as PI and submitting.
- Have your postdocs write your grant proposals for you. In their free time. Don't bother reading them, just have your secretary put your name on as PI and submit. They remain employed as long as they bring in the money. Keep the best grant writers convinced that they need to publish a lot more before they'll be ready to move on. Find ways to delay publication of their data. Sit back and enjoy.
9) When you first arrived at your University your Department purchased a confocal microscope you needed. The deal was that you would be in charge of this instrument, but that others in the Department would have access and you would provide the training. Over the years you have been doing less and less confocal work, to the point that you now no longer need such an instrument. What do you do?
- Still put in the effort to keep the instrument maintained, and to train other users.
- Retain control over the instrument, but leave the training to the tech in your lab who has used it once or twice.
- Tell the Department that you no longer have any need for the instrument and that you refuse to be responsible for it anymore.
- Sell the confocal microscope on Ebay and use the proceeds to pay for a skiing vacation in the Alps.
10) A young, completely unknown, recently graduated PhD from Estonia emails you a rough draft of a manuscript for you comments. Upon reading the manuscript you realize that this is Nobel-prize stuff. What do you do?
- Go through the manuscript carefully, provide thoughtful and detailed comments, and inform the PhD as to the impact the work will have.
- Offer to collaborate with the PhD, but insist they will be senior author.
- Tell the PhD that the manuscript is interesting, but needs further experiments. Offer them a postdoctoral position in your lab so they can do the work with you, and that you will then publish the work with you as senior author.
- Email the PhD telling them the work is crap not worthy of submission anywhere. Meanwhile have everyone in your lab do some quick and easy experiments you can add to the existing manuscript, rewrite it and submit.
Answer Key:
For each "a" give yourself one point, for each "b" two points, each "c" three points, and each "d" four points.
If you scored:
10-20
Sorry, you are just too nice to ever be a Rockstar Scientist!!!!! Your colleagues really like you and you are certainly popular among the grad students. But stardom will forever be out of reach.
21-30
Not going to make it to Rockstar Scientist!!!!! You try, but that pesky conscience keeps getting in the way.
31-39
You're almost a Rockstar Scientist!!!!! But not quite. What's holding you back? Couldn't sell the confocal? Missed the plane to Stockholm?
40
You are truly a Rockstar Scientist!!!!! Congratulations! It must be great not having to write grant proposals anymore. What did you get for the confocal and how was Stockholm?
Happy Non-Denominational Festive Season!
Monday, December 15, 2008
Numbers
Random observations involving numbers...
Sometime very early this morning my blog had it's 7000th visitor (someone from Brisbane, Australia - bet the weather's better there than here). It blows me away that people not only read some of my blather in the first place, but come back for more! Hat tip to DrugMonkey - much of the traffic I'm getting is due to him.
My very first experimental work-based paper (I used to be a computational biophysicist) has just hit 80 citations. That's about 10/year. Apparently an old dog can teach himself new tricks...
30. The number of degrees (in F; that's about 17C) the temperature is expected to drop by between the time I arrived at work this morning and the time I leave this evening. We're expecting 2-4 inches of mixed snow and ice this evening. Snow I like. Ice sucks.
Sometime very early this morning my blog had it's 7000th visitor (someone from Brisbane, Australia - bet the weather's better there than here). It blows me away that people not only read some of my blather in the first place, but come back for more! Hat tip to DrugMonkey - much of the traffic I'm getting is due to him.
My very first experimental work-based paper (I used to be a computational biophysicist) has just hit 80 citations. That's about 10/year. Apparently an old dog can teach himself new tricks...
30. The number of degrees (in F; that's about 17C) the temperature is expected to drop by between the time I arrived at work this morning and the time I leave this evening. We're expecting 2-4 inches of mixed snow and ice this evening. Snow I like. Ice sucks.
Monday, December 08, 2008
So you want a piece of the NSF pie? Some thoughts.
Some random thoughts to wrap up this series.
As before, disclaimers here.
1) The review process-
The NSF review process is very different to that of the NIH. I gather the different Directorates, and even organizations within Directorates, have some leeway in deciding how the review process works. What I'm about to describe is the process my own proposals go through at the BIO Directorate. Your mileage may vary.
Review is basically a two-stage process. First your proposal is sent out to a bunch (at least two, sometimes up to six or seven) ad hoc reviewers, plus at least two members of the review panel. These reviewers send in their reviews via Fastlane. The second step is the review panel. The panel considers all of the reviews (as many as eight or even nine), discusses the proposal and assigns a rank (not a score). The Program Director then takes the rankings and figures out what she can fund.
So you get a bunch of reviews. This is good and bad. On the plus side, you can survive a mediocre review if all the others are stellar. You also get a lot of (hopefully) useful feedback. On the minus side, if one of the panel members who reviewed your grant really didn't like it, your odds of being funded are not great even if all the ad hoc reviewers loved your proposal. As with NIH study sections, you need an advocate on the review panel.
2) Proposal rankings-
As I noted above, reviewers don't score NSF proposals per se. They give them a ranking. The reviewers assign a rank of Excellent, Very Good, Good, Fair or Poor (from memory). The review panel takes those and assigns an aggregate rank of Outstanding, Highly Meritorious, Meritorious, or Non-Competitive. The last two categories won't be funded. Sadly, it is often the case that not all of the proposals ranked as Outstanding can be funded...
WARNING! WARNING! ANECDOTAL STUFF FOLLOWS!
Okay, the following stuff is purely anecdotal - you've been warned. I'm posting it because I've heard the same things from multiple people (n > 6), leading me to think there might just be a grain of truth in each of these...
3) NSF doesn't want young people to fail-
NSF Program Directors have a lot of say over who gets funded and who doesn't. Clearly they can't over-ride the reviewers to much or too many times otherwise they would have a very hard time finding people willing to review proposals. But they can nudge proposals over the funding line if they feel it's warranted. What I have heard from multiple people is that Program Directors will sometimes do this in order to fund an otherwise unfunded young investigator on the verge of their tenure decision. Of course said investigator needs to have been productive enough to warrant this.
This is not something I would bet any money on. Or my tenure. It's obviously best to not to have to rely upon the largesse of a Program Director.
4) Once you're in you're golden-
By that I mean once you're funded by the NSF, as long as you're productive, your Program Director will try to ensure you maintain the ability to be productive. I am by no means suggesting that the Program Director will fund your proposals over higher scoring proposals. Rather that if you're on the funding edge and it's a choice between you as a previously NSF-funded, productive PI and someone else, you will get the nod (remember - this is anecdotal stuff).
I have brought this up hesitantly because it sounds like a mechanism for maintaining the status quo. I would like to think that's not the intent, rather that it's more a matter of Program Directors being loyal to their productive charges. Can it have the side effect of maintaining the status quo? Yes. The someone else is likely a new investigator... Please note that a) THIS IS ANECDOTAL, b) this is somewhat at odds with the previous anecdotal point in this post, and c) I'm not defending this.
As before, disclaimers here.
1) The review process-
The NSF review process is very different to that of the NIH. I gather the different Directorates, and even organizations within Directorates, have some leeway in deciding how the review process works. What I'm about to describe is the process my own proposals go through at the BIO Directorate. Your mileage may vary.
Review is basically a two-stage process. First your proposal is sent out to a bunch (at least two, sometimes up to six or seven) ad hoc reviewers, plus at least two members of the review panel. These reviewers send in their reviews via Fastlane. The second step is the review panel. The panel considers all of the reviews (as many as eight or even nine), discusses the proposal and assigns a rank (not a score). The Program Director then takes the rankings and figures out what she can fund.
So you get a bunch of reviews. This is good and bad. On the plus side, you can survive a mediocre review if all the others are stellar. You also get a lot of (hopefully) useful feedback. On the minus side, if one of the panel members who reviewed your grant really didn't like it, your odds of being funded are not great even if all the ad hoc reviewers loved your proposal. As with NIH study sections, you need an advocate on the review panel.
2) Proposal rankings-
As I noted above, reviewers don't score NSF proposals per se. They give them a ranking. The reviewers assign a rank of Excellent, Very Good, Good, Fair or Poor (from memory). The review panel takes those and assigns an aggregate rank of Outstanding, Highly Meritorious, Meritorious, or Non-Competitive. The last two categories won't be funded. Sadly, it is often the case that not all of the proposals ranked as Outstanding can be funded...
WARNING! WARNING! ANECDOTAL STUFF FOLLOWS!
Okay, the following stuff is purely anecdotal - you've been warned. I'm posting it because I've heard the same things from multiple people (n > 6), leading me to think there might just be a grain of truth in each of these...
3) NSF doesn't want young people to fail-
NSF Program Directors have a lot of say over who gets funded and who doesn't. Clearly they can't over-ride the reviewers to much or too many times otherwise they would have a very hard time finding people willing to review proposals. But they can nudge proposals over the funding line if they feel it's warranted. What I have heard from multiple people is that Program Directors will sometimes do this in order to fund an otherwise unfunded young investigator on the verge of their tenure decision. Of course said investigator needs to have been productive enough to warrant this.
This is not something I would bet any money on. Or my tenure. It's obviously best to not to have to rely upon the largesse of a Program Director.
4) Once you're in you're golden-
By that I mean once you're funded by the NSF, as long as you're productive, your Program Director will try to ensure you maintain the ability to be productive. I am by no means suggesting that the Program Director will fund your proposals over higher scoring proposals. Rather that if you're on the funding edge and it's a choice between you as a previously NSF-funded, productive PI and someone else, you will get the nod (remember - this is anecdotal stuff).
I have brought this up hesitantly because it sounds like a mechanism for maintaining the status quo. I would like to think that's not the intent, rather that it's more a matter of Program Directors being loyal to their productive charges. Can it have the side effect of maintaining the status quo? Yes. The someone else is likely a new investigator... Please note that a) THIS IS ANECDOTAL, b) this is somewhat at odds with the previous anecdotal point in this post, and c) I'm not defending this.
Friday, December 05, 2008
So you want a piece of the NSF pie? Broader Impacts.
As before, necessary disclaimers can be found here.
The dreaded Broader Impacts... This is the place many of the proposals I've reviewed have significant weaknesses. It used to be you could just pay lip service to these. Half a page max at the end of the proposal would be plenty. Not anymore. The NSF has instructed its reviewers to take these very seriously. And I can assure you the Program Officers take them very, very seriously. So when you write a proposal destined for the NSF, you need to take the broader impacts very seriously.
Let's start with why NSF requires broader impacts - understanding this can help formulate some for your proposal. It is important to understand that the NSF will not give you a grant just to do research. Read their mission statement:
To promote the progress of science; to advance the national health, prosperity and welfare; to secure the national defense (NSF Act of 1950).
Note that it's promote the progress of science. That's much more than just funding research. The NSF's mission includes science education and training, and dissemination of scientific knowledge to the broader population. That's where the broader impacts criteria come from.
So what are broader impacts? The place to start is with the NSF Grant Proposal Guide. Find the section on review criteria and specifically that part dealing with the broader impacts. As you will see these are split into two categories:
Basically it comes down to how you're going to tie education and training into your research activities and how you're going to go about improving diversity within science. The NSF has kindly put together a crib sheet describing how you might address these. It can be downloaded via at www.nsf.gov/pubs/2002/nsf022/bicexamples.pdf (for some reason I can't get Blogger to publish this as a link...). These are just suggestions and nobody in their right mind would propose to tackle all of them. Pick and choose those that work for your circumstances. Be creative and come up with new ones.
Here are some of my thoughts on how to address the broader impacts.
Integration of research and education:
This is the easier of the two criteria to deal with. It's important to understand here that "education" includes training, teaching and dissemination.
Training- In your proposal you want to talk about how you're going to involve trainees in your research. Trainees can include undergrads, grad students and postdocs. You could just say they will be involved, but it's much better to provide specific examples of how they will be involved. For example, what pieces of your project would be suitable fodder for undergrad researchers? Some verbiage describing how your trainees will be trained can also help.
Involving undergrads (and possibly high school students) is a very good thing to do (at least in NSF's view). And can be very rewarding for the PI as well. If you can, write an undergrad or two into each budget year. And specifically state that you will actively encourage even more undergrads to join your lab and that you will apply for REU supplements to support them (read up on these - it's easy money and really looks good when you go to renew your NSF grant). And don't forget to tout all the great work you've done with undergrads in the past.
Teaching- You can keep this as simple as stating that you will integrate the results of your research into your teaching (about the minimum in terms of addressing this, and really all I do), through to getting involved in teaching at K-12 schools. Another possibility is to host high school science teachers over the summer. The NSF has a whole program devoted to funding this kind of thing.
Dissemination- It's a given that you're going to publish and present your data at meetings etc. The difference here is that you need to explicitly state that. And describe how your trainees are going to be disseminating as well. Don't forget to budget funds for these activities, including funds to send trainees to meetings. It's important to back these things up with a real commitment - money.
If you have other opportunities to disseminate the results of your research (e.g. you've been invited to write a review or book chapter), talk about those. The book I recently edited is broader impact/dissemination fodder I count tout. Are you depositing stuff in publically-accessible databases? That's more dissemination stuff.
Diversity:
You can address this separately from the above, or integrated within it. Either works.
What the NSF wants here is some description of how you are going to try to involve people from traditionally under-represented (in science) groups in your research program. I've always found this the most difficult to address. You want to write something that you actually have a chance to succeeding at. If you read the NSF Broader Impacts crib sheet (www.nsf.gov/pubs/2002/nsf022/bicexamples.pdf) you can get some reasonable ideas. Collaborations with PI's who are, or who work with under-represented people count. As do collaborations with faculty at four-year colleges. The ultimate is to have members of under-represented groups working in your lab.
If you have any kind of track record of doing any of this, tout it loudly and clearly.
One last word of advice. Addressing the broader impacts sufficiently is going to require valuable real estate in your 15 page proposal. I don't know how much would be considered too much, but I can tell you less than a page will likely doom your grant. In my last two NSF proposals (both renewals, one that was funded in 2004 and one that has just been recommended for funding) I used at about two pages (not counting the space used to describe the broader impact work done with the prior period of funding).
The dreaded Broader Impacts... This is the place many of the proposals I've reviewed have significant weaknesses. It used to be you could just pay lip service to these. Half a page max at the end of the proposal would be plenty. Not anymore. The NSF has instructed its reviewers to take these very seriously. And I can assure you the Program Officers take them very, very seriously. So when you write a proposal destined for the NSF, you need to take the broader impacts very seriously.
Let's start with why NSF requires broader impacts - understanding this can help formulate some for your proposal. It is important to understand that the NSF will not give you a grant just to do research. Read their mission statement:
Note that it's promote the progress of science. That's much more than just funding research. The NSF's mission includes science education and training, and dissemination of scientific knowledge to the broader population. That's where the broader impacts criteria come from.
So what are broader impacts? The place to start is with the NSF Grant Proposal Guide. Find the section on review criteria and specifically that part dealing with the broader impacts. As you will see these are split into two categories:
- Integration of Research and Education
- Integrating Diversity into NSF Programs, Projects, and Activities
Basically it comes down to how you're going to tie education and training into your research activities and how you're going to go about improving diversity within science. The NSF has kindly put together a crib sheet describing how you might address these. It can be downloaded via at www.nsf.gov/pubs/2002/nsf022/bicexamples.pdf (for some reason I can't get Blogger to publish this as a link...). These are just suggestions and nobody in their right mind would propose to tackle all of them. Pick and choose those that work for your circumstances. Be creative and come up with new ones.
Here are some of my thoughts on how to address the broader impacts.
Integration of research and education:
This is the easier of the two criteria to deal with. It's important to understand here that "education" includes training, teaching and dissemination.
Training- In your proposal you want to talk about how you're going to involve trainees in your research. Trainees can include undergrads, grad students and postdocs. You could just say they will be involved, but it's much better to provide specific examples of how they will be involved. For example, what pieces of your project would be suitable fodder for undergrad researchers? Some verbiage describing how your trainees will be trained can also help.
Involving undergrads (and possibly high school students) is a very good thing to do (at least in NSF's view). And can be very rewarding for the PI as well. If you can, write an undergrad or two into each budget year. And specifically state that you will actively encourage even more undergrads to join your lab and that you will apply for REU supplements to support them (read up on these - it's easy money and really looks good when you go to renew your NSF grant). And don't forget to tout all the great work you've done with undergrads in the past.
Teaching- You can keep this as simple as stating that you will integrate the results of your research into your teaching (about the minimum in terms of addressing this, and really all I do), through to getting involved in teaching at K-12 schools. Another possibility is to host high school science teachers over the summer. The NSF has a whole program devoted to funding this kind of thing.
Dissemination- It's a given that you're going to publish and present your data at meetings etc. The difference here is that you need to explicitly state that. And describe how your trainees are going to be disseminating as well. Don't forget to budget funds for these activities, including funds to send trainees to meetings. It's important to back these things up with a real commitment - money.
If you have other opportunities to disseminate the results of your research (e.g. you've been invited to write a review or book chapter), talk about those. The book I recently edited is broader impact/dissemination fodder I count tout. Are you depositing stuff in publically-accessible databases? That's more dissemination stuff.
Diversity:
You can address this separately from the above, or integrated within it. Either works.
What the NSF wants here is some description of how you are going to try to involve people from traditionally under-represented (in science) groups in your research program. I've always found this the most difficult to address. You want to write something that you actually have a chance to succeeding at. If you read the NSF Broader Impacts crib sheet (www.nsf.gov/pubs/2002/nsf022/bicexamples.pdf) you can get some reasonable ideas. Collaborations with PI's who are, or who work with under-represented people count. As do collaborations with faculty at four-year colleges. The ultimate is to have members of under-represented groups working in your lab.
If you have any kind of track record of doing any of this, tout it loudly and clearly.
One last word of advice. Addressing the broader impacts sufficiently is going to require valuable real estate in your 15 page proposal. I don't know how much would be considered too much, but I can tell you less than a page will likely doom your grant. In my last two NSF proposals (both renewals, one that was funded in 2004 and one that has just been recommended for funding) I used at about two pages (not counting the space used to describe the broader impact work done with the prior period of funding).
Wednesday, December 03, 2008
So you want a piece of the NSF pie? Scope and budget.
Herein I continue my somewhat disorganized discussion of NSF funding...
Necessary disclaimers can be found here.
Okay, so now you've had a chance to study NSF's award statistics, it's time to think about the scope of an NSF proposal. Let's face it, a five year, four/five specific aim proposal probably isn't going to fly. Most proposals funded by the NSF appear to be for around three years and have a modest budget. Poke around here to see what the NSF has funded in your area recently to get a better idea of what you're dealing with.
The budget is all important here - it will define the scope of what you propose to do. Remember, those median annual award sizes are totals - direct plus indirect costs. Obviously if you are at an institution with a high F&A rate (say 90%), you're going to be able to ask for more than the median amounts. But don't get greedy - the NSF has limited funds. If your F&A rate is more modest (say around 50%), think in terms of the median amount. Figure out what the median direct rate would be - that's roughly what you'll have to work with (in practice, if you're funded, you're likely to take a modest budget cut, but don't worry about that too much). Let's say you're applying to the BIO directorate, MCB organization, Molecular Biophysics section (one of the more generous sections and the one I'm funded through), and have a F&A rate of 50%. The median direct costs (using the 2007 median annual award) are then:
Directs = $159,113/1.5 = $106,075
Basically $100k/year for three (maybe four) years in this example (remember - your mileage will vary). So the important question becomes what can you do with that? The answer will define the scope of your proposal.
What you can do with that amount of money is highly dependent upon the kind of work you do and the type of position you have. I'm in a college of medicine where each PI pays the stipend and benefits (but not tuition - yet) of their graduate students (no TA's). I might just be able to pay two students based on an award of $100/k per year if I'm very, very careful (i.e. miserly). Or one postdoc and part of a tech's salary. Etc. These considerations clearly limit the scope of what I can propose to do with an NSF grant. In the end I've found two specific aims are about all I can manage. If you're in a setting where TAships for grad students are plentiful, perhaps you can manage more.
Something to keep in mind while we're on the topic of the budget. The NSF will only allow you to pay two months of your salary per year from all sources of NSF funds you have access to. Recently there was a discussion at DrugMonkey's place about this. Let's take me as an example. I'm PI on an NSF research grant and on an NSF REU Site grant. If I get 0.5 months salary from the REU grant I can only take 1.5 months salary from the research grant. NSF doesn't appear to care how much of your salary comes from grants from other agencies.
One last thing while I'm thinking about it. Should you get an NSF award (congrats!), there are two types of research grant; Standard and Continuing. You have no say in which one you get. If you're awarded a Standard grant you're given all the money up front. If you get a Continuing grant you get one year's money at a time. Each new year is contingent on an annual report you have to submit (you have to put these in with Standard grants too). I've always had a Continuing grant and have never had an issue with obtaining the next year's money (I am quite diligent about putting in the required report). I've also never heard of anyone not getting their next year's money, but I suppose it's a possibility.
Necessary disclaimers can be found here.
Okay, so now you've had a chance to study NSF's award statistics, it's time to think about the scope of an NSF proposal. Let's face it, a five year, four/five specific aim proposal probably isn't going to fly. Most proposals funded by the NSF appear to be for around three years and have a modest budget. Poke around here to see what the NSF has funded in your area recently to get a better idea of what you're dealing with.
The budget is all important here - it will define the scope of what you propose to do. Remember, those median annual award sizes are totals - direct plus indirect costs. Obviously if you are at an institution with a high F&A rate (say 90%), you're going to be able to ask for more than the median amounts. But don't get greedy - the NSF has limited funds. If your F&A rate is more modest (say around 50%), think in terms of the median amount. Figure out what the median direct rate would be - that's roughly what you'll have to work with (in practice, if you're funded, you're likely to take a modest budget cut, but don't worry about that too much). Let's say you're applying to the BIO directorate, MCB organization, Molecular Biophysics section (one of the more generous sections and the one I'm funded through), and have a F&A rate of 50%. The median direct costs (using the 2007 median annual award) are then:
Basically $100k/year for three (maybe four) years in this example (remember - your mileage will vary). So the important question becomes what can you do with that? The answer will define the scope of your proposal.
What you can do with that amount of money is highly dependent upon the kind of work you do and the type of position you have. I'm in a college of medicine where each PI pays the stipend and benefits (but not tuition - yet) of their graduate students (no TA's). I might just be able to pay two students based on an award of $100/k per year if I'm very, very careful (i.e. miserly). Or one postdoc and part of a tech's salary. Etc. These considerations clearly limit the scope of what I can propose to do with an NSF grant. In the end I've found two specific aims are about all I can manage. If you're in a setting where TAships for grad students are plentiful, perhaps you can manage more.
Something to keep in mind while we're on the topic of the budget. The NSF will only allow you to pay two months of your salary per year from all sources of NSF funds you have access to. Recently there was a discussion at DrugMonkey's place about this. Let's take me as an example. I'm PI on an NSF research grant and on an NSF REU Site grant. If I get 0.5 months salary from the REU grant I can only take 1.5 months salary from the research grant. NSF doesn't appear to care how much of your salary comes from grants from other agencies.
One last thing while I'm thinking about it. Should you get an NSF award (congrats!), there are two types of research grant; Standard and Continuing. You have no say in which one you get. If you're awarded a Standard grant you're given all the money up front. If you get a Continuing grant you get one year's money at a time. Each new year is contingent on an annual report you have to submit (you have to put these in with Standard grants too). I've always had a Continuing grant and have never had an issue with obtaining the next year's money (I am quite diligent about putting in the required report). I've also never heard of anyone not getting their next year's money, but I suppose it's a possibility.
Monday, December 01, 2008
So you want a piece of the NSF pie? The NSF is NOT the NIH.
I've decided to put down some thoughts of mine regarding obtaining funding from the NSF. This will likely end up being a series of posts...
First, the necessary disclaimer: These are my opinions only, shaped by my experiences as an NSF-funded investigator (in a College of Medicine), a reviewer of NSF proposals, plus the experiences of some friends and collaborators. Your mileage will almost certainly vary.
Second, some background: I was awarded my first NSF grant in 2001. I renewed that in 2004 and have recently heard that my second renewal has been recommended for funding. I am also PI on an NSF REU Site grant. I have reviewed many NSF proposals, most from the BIO directorate and some from MPS, on an ad hoc basis. Finally, I sat on an NSF SBIR review panel for two years.
Now, let's get down to business here. If you're in the biological sciences, please repeat after me:
Keep repeating this until you are firmly convinced of the truth in that statement.
Now you would think that that's obvious, right? Apparently it isn't to about 25% of the applicants whose proposals I've reviewed for the BIO directorate...
Here are some, but not all, major differences:
1) Budget-
The projected FY 2009 NIH budget is $28.7 billion for R&D expenditures. The NSF? $4.5 billion for R&D. This is under the current Continuing Resolution. See the AAAS September R&D Update to see where I got these figures. These numbers have a very profound effect on many aspects of NSF funding versus NIH...
2) Mission-
The mission of the NIH is "science in pursuit of fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to extend healthy life and reduce the burdens of illness and disability."
The mission of the NSF "includes support for all fields of fundamental science and engineering, except for medical sciences." [Emphasis mine.]
The NIH funds biomedical research. The NSF funds all of the other sciences. Including social. Plus engineering. And mathematics. And the NSF mission includes a strong science education component (which I believe accounts for the disparity between the $6 billion total budget of the NSF versus the $4.5 billion it spends on R&D).
So, don't go proposing to solve cancer, or heart disease, or Alzheimer's in your NSF proposal. If you do I can pretty much guarantee it won't be funded. Basic biological science is fine (otherwise I wouldn't be funded), so there is some overlap between the NSF and NIH. But you need to be careful about how you sell your basic science to the NSF. Can't figure out how to sell your science without referring to the medical benefits of the research? Don't bother applying to the NSF.
3) Scope-
The scope (budget and period of funding) of an NSF proposal is very different to that of an NIH R01. The NSF keeps some interesting funding statistics. If you're considering applying to the NSF, go find the data for the last year or two for the directorate and organization (within the directorate) you are most likely going to send your proposal to. Study those numbers. Not the funding rates. The mean award duration and median annual award size. Your proposal needs to be in the same ballpark as those numbers (unless you're applying to a special program with special budget instructions). Sure, you can ask for more time and/or money, but you'd better do a very, very good job of justifying why you need more.
Oh yeah, one other thing. Those median annual award sizes? That's total. Direct plus indirect costs.
4) Proposals-
Aside from the fact that an NSF proposal is limited to 15 pages versus the (current) 25 page limit at NIH, there are other differences. I'm not going to go into great detail here. I'll probably cover some of that stuff in later posts. The main thing is to go study the NSF review criteria. And take them seriously. Intellectual Merit pretty much refers to the quality of the science proposed (and of the proposer). That Broader Impact stuff? Not something the NIH puts much weight on. The NSF, on the other hand, is very serious about it. I'm going to write a post devoted purely to Broader Impacts since this is where I've seen many proposals fail. Particularly those form new investigators.
5) Effect on tenure-
This is aimed at those of you in a College of Medicine. I got tenure based on my NSF funding. Here that's considered sufficient to satisfy the funding part of the tenure equation (assuming you've done very well on all other parts of the equation). That is not the case in all medical schools. There are Deans of medical schools who consider NSF money second class at best and will not approve tenure for someone with only NSF funding. Be sure you know what the ground rules are at your institution.
First, the necessary disclaimer: These are my opinions only, shaped by my experiences as an NSF-funded investigator (in a College of Medicine), a reviewer of NSF proposals, plus the experiences of some friends and collaborators. Your mileage will almost certainly vary.
Second, some background: I was awarded my first NSF grant in 2001. I renewed that in 2004 and have recently heard that my second renewal has been recommended for funding. I am also PI on an NSF REU Site grant. I have reviewed many NSF proposals, most from the BIO directorate and some from MPS, on an ad hoc basis. Finally, I sat on an NSF SBIR review panel for two years.
Now, let's get down to business here. If you're in the biological sciences, please repeat after me:
The NSF is NOT the NIH.
Keep repeating this until you are firmly convinced of the truth in that statement.
Now you would think that that's obvious, right? Apparently it isn't to about 25% of the applicants whose proposals I've reviewed for the BIO directorate...
Here are some, but not all, major differences:
1) Budget-
The projected FY 2009 NIH budget is $28.7 billion for R&D expenditures. The NSF? $4.5 billion for R&D. This is under the current Continuing Resolution. See the AAAS September R&D Update to see where I got these figures. These numbers have a very profound effect on many aspects of NSF funding versus NIH...
2) Mission-
The mission of the NIH is "science in pursuit of fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to extend healthy life and reduce the burdens of illness and disability."
The mission of the NSF "includes support for all fields of fundamental science and engineering, except for medical sciences." [Emphasis mine.]
The NIH funds biomedical research. The NSF funds all of the other sciences. Including social. Plus engineering. And mathematics. And the NSF mission includes a strong science education component (which I believe accounts for the disparity between the $6 billion total budget of the NSF versus the $4.5 billion it spends on R&D).
So, don't go proposing to solve cancer, or heart disease, or Alzheimer's in your NSF proposal. If you do I can pretty much guarantee it won't be funded. Basic biological science is fine (otherwise I wouldn't be funded), so there is some overlap between the NSF and NIH. But you need to be careful about how you sell your basic science to the NSF. Can't figure out how to sell your science without referring to the medical benefits of the research? Don't bother applying to the NSF.
3) Scope-
The scope (budget and period of funding) of an NSF proposal is very different to that of an NIH R01. The NSF keeps some interesting funding statistics. If you're considering applying to the NSF, go find the data for the last year or two for the directorate and organization (within the directorate) you are most likely going to send your proposal to. Study those numbers. Not the funding rates. The mean award duration and median annual award size. Your proposal needs to be in the same ballpark as those numbers (unless you're applying to a special program with special budget instructions). Sure, you can ask for more time and/or money, but you'd better do a very, very good job of justifying why you need more.
Oh yeah, one other thing. Those median annual award sizes? That's total. Direct plus indirect costs.
4) Proposals-
Aside from the fact that an NSF proposal is limited to 15 pages versus the (current) 25 page limit at NIH, there are other differences. I'm not going to go into great detail here. I'll probably cover some of that stuff in later posts. The main thing is to go study the NSF review criteria. And take them seriously. Intellectual Merit pretty much refers to the quality of the science proposed (and of the proposer). That Broader Impact stuff? Not something the NIH puts much weight on. The NSF, on the other hand, is very serious about it. I'm going to write a post devoted purely to Broader Impacts since this is where I've seen many proposals fail. Particularly those form new investigators.
5) Effect on tenure-
This is aimed at those of you in a College of Medicine. I got tenure based on my NSF funding. Here that's considered sufficient to satisfy the funding part of the tenure equation (assuming you've done very well on all other parts of the equation). That is not the case in all medical schools. There are Deans of medical schools who consider NSF money second class at best and will not approve tenure for someone with only NSF funding. Be sure you know what the ground rules are at your institution.
Tuesday, November 25, 2008
Book amusement II
Earlier this year a book I edited was published. I had earlier posted about some amusing keywords listed on Amazon.com for it. This morning I checked on Amazon.com to see if anyone there had written a review of it. No one had. But what I did find there did give me yet another chuckle...
Now, if you buy this book directly from the publisher you'll pay ~$71. Amazon doesn't sell the book directly, but currently offers it via four resellers plus a used book seller. Here are the listed prices:
Reseller 1- $42.00
Reseller 2- $64.65
Reseller 3- $93.89
Reseller 4- $107.85
Almost a three-fold range!
Used book dealer- $112.22!!!!!! And yes, this is listed as a used copy... Perhaps one originally bought from International Books...
Now, if you buy this book directly from the publisher you'll pay ~$71. Amazon doesn't sell the book directly, but currently offers it via four resellers plus a used book seller. Here are the listed prices:
Reseller 1- $42.00
Reseller 2- $64.65
Reseller 3- $93.89
Reseller 4- $107.85
Almost a three-fold range!
Used book dealer- $112.22!!!!!! And yes, this is listed as a used copy... Perhaps one originally bought from International Books...
Wednesday, November 19, 2008
It's alive!!!!!
Yesterday I received a rather unexpected call - from my NSF program director. Apparently he's recommending that my recent proposal be funded! If you recall, this proposal is based on a new research program I was trying to raise from the ashes of my mid-tenure crisis. Apparently that new research program now has a life.
Odyssey's new research program lurches into existence.
It's alive!
This certainly changes things over the next couple of months. Instead of working frantically to generate more preliminary data for a resubmission of the proposal, I get to look for a new postdoc.
And start up another new project.
'Cos my new research program needs...
A bride!
Odyssey's new research program lurches into existence.
It's alive!
This certainly changes things over the next couple of months. Instead of working frantically to generate more preliminary data for a resubmission of the proposal, I get to look for a new postdoc.
And start up another new project.
'Cos my new research program needs...
A bride!
Friday, November 14, 2008
Five things meme...
Gee, thanks JLK...
5 things I was doing 10 years ago:
1. Wondering if I should add experimental work to my research (I was a computational biologist).
2. Being totally besotted with my baby daughter.
3. Realizing the tenure clock ticks very, very quickly.
4. Enjoying being a first-time homeowner.
5. Celebrating the publication of my first paper as a tenure-track faculty (actually ten years ago this week!).
5 things on my to do list today:
1. Finish reviewing the all-too-large pile of travel grant applications on my desk.
2. Plan out how I'm going to squeeze in a protein expression and purification (that I'll be doing myself) in the next two or three weeks. My work has morphed over ten years from purely computational to almost purely experimental...
3. Spend time with my family.
4. Try to avoid spending an excessive amount of time reading blogs.
5. Go to bed at a decent hour.
5 snacks I love:
1. Apples.
2. Potato chips.
3. White chocolate chip-macademia nut cookies.
4. A freshly-baked cheese danish or walnut-jam scone or cinnamon doughnut from the bakery down the street (okay, so I cheated a little and squeezed three into one).
5. Beer (where I come from beer qualifies as a snack. And breakfast, lunch and dinner).
5 things I would do if I were a millionaire:
1. Put away a bunch for the kid's college tuitions.
2. Take a family vacation to Australia.
3. Take Mrs. Odyssey to Europe (sans kids).
4. Pay off the house.
5. Renovate the kitchen.
5 places I've lived:
1. Australia.
2. Pennsylvania.
3. North Carolina.
4. Missouri.
5. Maryland.
5 jobs I've had:
1. Selling car parts.
2. Chemistry lab demonstrator (like a TA).
3. Lead chemist at a phosphate fertilizer plant (thankfully only temporarily, although the town it is based in is one of the most beautiful locales I've ever seen).
4. Postdoctoral fellow.
5. Research Associate (this was the title the institution I was at used for postdocs who'd been around for too long).
5 people I'll tag:
Five people I don't know enough about (or want to know more about).
1. DocStymie.
2. JollyRgr (surprise we with things I don't know).
3. Professor in Training (if you can fit a meme in with everything else you've got going).
4. Bugger it. I'm stopping at three.
The 5 Things Meme
5 things I was doing 10 years ago:
1. Wondering if I should add experimental work to my research (I was a computational biologist).
2. Being totally besotted with my baby daughter.
3. Realizing the tenure clock ticks very, very quickly.
4. Enjoying being a first-time homeowner.
5. Celebrating the publication of my first paper as a tenure-track faculty (actually ten years ago this week!).
5 things on my to do list today:
1. Finish reviewing the all-too-large pile of travel grant applications on my desk.
2. Plan out how I'm going to squeeze in a protein expression and purification (that I'll be doing myself) in the next two or three weeks. My work has morphed over ten years from purely computational to almost purely experimental...
3. Spend time with my family.
4. Try to avoid spending an excessive amount of time reading blogs.
5. Go to bed at a decent hour.
5 snacks I love:
1. Apples.
2. Potato chips.
3. White chocolate chip-macademia nut cookies.
4. A freshly-baked cheese danish or walnut-jam scone or cinnamon doughnut from the bakery down the street (okay, so I cheated a little and squeezed three into one).
5. Beer (where I come from beer qualifies as a snack. And breakfast, lunch and dinner).
5 things I would do if I were a millionaire:
1. Put away a bunch for the kid's college tuitions.
2. Take a family vacation to Australia.
3. Take Mrs. Odyssey to Europe (sans kids).
4. Pay off the house.
5. Renovate the kitchen.
5 places I've lived:
1. Australia.
2. Pennsylvania.
3. North Carolina.
4. Missouri.
5. Maryland.
5 jobs I've had:
1. Selling car parts.
2. Chemistry lab demonstrator (like a TA).
3. Lead chemist at a phosphate fertilizer plant (thankfully only temporarily, although the town it is based in is one of the most beautiful locales I've ever seen).
4. Postdoctoral fellow.
5. Research Associate (this was the title the institution I was at used for postdocs who'd been around for too long).
5 people I'll tag:
Five people I don't know enough about (or want to know more about).
1. DocStymie.
2. JollyRgr (surprise we with things I don't know).
3. Professor in Training (if you can fit a meme in with everything else you've got going).
4. Bugger it. I'm stopping at three.
Wednesday, November 12, 2008
A week in the life of Dr. Odyssey
My last post was on having undergraduates do research. In the comments JLK asked:
Why be a professor at an MRU if you're not going to do your own research?
A simple question with a rather complex answer (or answers). The short answer is stuff happens. Read on for the more detailed answer.
I suspect the vast majority of us start our tenure-track positions at MRU's (Major Research University) fully intending to literally do our own research. Stand side-by-side with our trainees at the bench sweating over our own experiments. That was certainly my intent.
For the first few years it can be like that. But over time things change. We obtain new and more responsibilities. Grant writing. Publishing. Teaching. Committees. Reviewing (both manuscripts and grant proposals).
As we become more successful and better known the list grows. Study sections/review panels for grantmaking agencies. Traveling to give talks at other institutions. Meetings/conferences. Perhaps editorial duties at a journal. Service within scientific/professional societies. Even more grant writing. If you're lucky enough to gain "rockstar" professor status you can end up with even more of these responsibilities.
The time for actually doing our own research dwindles. I still manage to carve out some time to spend in the lab, but it's very limited. Instead we live vicariously through the efforts of the people in our groups. Instead of doing research we direct it. It's not that we don't want to do our own research. It's more that it's no longer in our job descriptions. By the time we hit mid-career (associate prof level - where I currently am) our job is to provide the environment, funding and intellectual background necessary for the research to occur.
Here is my week as an illustration (the current week, and just the weekdays). Note that my teaching load is very light compared to someone not in a College of Medicine.
Monday-
Morning: I got in late (mid-morning) after taking one of my daughters to the dentist... I am an officer in a subgroup within a scientific society. I spent over an hour Monday morning dealing with subgroup business. This was followed by time spent in the lab helping an undergrad debug an experiment. I updated my blog while eating lunch.
Afternoon: More subgroup business. Finished preparing an exam I was administering in a 2nd year graduate student-level class Tuesday morning. Spent an hour working on a manuscript. Spent an hour with a student asking questions regarding the exam. Attended a seminar given by a candidate for an open faculty position. Talked to my lab tech about the protein prep she was working on.
Tuesday-
Morning: Attended the chalk talk (future research presentation) given by faculty candidate. Proctored exam in grad-level class (we don't have TA's), met with the faculty candidate. Ate a sandwich before walking to other side of campus for...
Afternoon: Grad student committee meeting. Spent most of remainder of the day reviewing protocols submitted to the Institutional Biosafety Committee (IBC), of which I am a member. Talked to undergrad and lab tech about ongoing experiments.
Wednesday-
Morning: Continued work on manuscript. Planned out project for a new high school intern about to join the lab. Wrote this blog entry. Will walk to other side of campus for...
Afternoon: IBC meeting. This will probably take at least two hours. At least they feed us... Hope to work some more on manuscript. Check with lab tech on results of a gel. Meet with new high school intern.
Thursday-
Morning: Start reviewing student travel award proposals for scientific society. Teach in 2nd year grad student course. Continue reviewing travel award proposals (there are 36 to do). Talk with undergrads in the lab about their projects.
Afternoon: Maybe spend two hours in the lab doing some research! Hopefully also work with lab tech learning new technique. Review more travel award proposals. Attend student seminar.
Friday-
Morning: I try to spend Friday mornings working at home, mostly writing and reviewing. I'll try to get a manuscript review done and finish the travel award reviewing.
Afternoon: Meet with people in the lab. Attend weekly meeting held by a Center I'm part of. Get back to work on that manuscript.
Okay, so maybe I don't have 36 travel award proposals to review every week. I may have more manuscripts or grant proposals to review. Or other committee meetings. Or travel. And I haven't detailed any of the constant interruptions that occur during each day. You can see it's difficult to carve out big blocks of time to spend in the lab during the week.
Such is life.
I wouldn't trade it for anything.
Why be a professor at an MRU if you're not going to do your own research?
A simple question with a rather complex answer (or answers). The short answer is stuff happens. Read on for the more detailed answer.
I suspect the vast majority of us start our tenure-track positions at MRU's (Major Research University) fully intending to literally do our own research. Stand side-by-side with our trainees at the bench sweating over our own experiments. That was certainly my intent.
For the first few years it can be like that. But over time things change. We obtain new and more responsibilities. Grant writing. Publishing. Teaching. Committees. Reviewing (both manuscripts and grant proposals).
As we become more successful and better known the list grows. Study sections/review panels for grantmaking agencies. Traveling to give talks at other institutions. Meetings/conferences. Perhaps editorial duties at a journal. Service within scientific/professional societies. Even more grant writing. If you're lucky enough to gain "rockstar" professor status you can end up with even more of these responsibilities.
The time for actually doing our own research dwindles. I still manage to carve out some time to spend in the lab, but it's very limited. Instead we live vicariously through the efforts of the people in our groups. Instead of doing research we direct it. It's not that we don't want to do our own research. It's more that it's no longer in our job descriptions. By the time we hit mid-career (associate prof level - where I currently am) our job is to provide the environment, funding and intellectual background necessary for the research to occur.
Here is my week as an illustration (the current week, and just the weekdays). Note that my teaching load is very light compared to someone not in a College of Medicine.
Monday-
Morning: I got in late (mid-morning) after taking one of my daughters to the dentist... I am an officer in a subgroup within a scientific society. I spent over an hour Monday morning dealing with subgroup business. This was followed by time spent in the lab helping an undergrad debug an experiment. I updated my blog while eating lunch.
Afternoon: More subgroup business. Finished preparing an exam I was administering in a 2nd year graduate student-level class Tuesday morning. Spent an hour working on a manuscript. Spent an hour with a student asking questions regarding the exam. Attended a seminar given by a candidate for an open faculty position. Talked to my lab tech about the protein prep she was working on.
Tuesday-
Morning: Attended the chalk talk (future research presentation) given by faculty candidate. Proctored exam in grad-level class (we don't have TA's), met with the faculty candidate. Ate a sandwich before walking to other side of campus for...
Afternoon: Grad student committee meeting. Spent most of remainder of the day reviewing protocols submitted to the Institutional Biosafety Committee (IBC), of which I am a member. Talked to undergrad and lab tech about ongoing experiments.
Wednesday-
Morning: Continued work on manuscript. Planned out project for a new high school intern about to join the lab. Wrote this blog entry. Will walk to other side of campus for...
Afternoon: IBC meeting. This will probably take at least two hours. At least they feed us... Hope to work some more on manuscript. Check with lab tech on results of a gel. Meet with new high school intern.
Thursday-
Morning: Start reviewing student travel award proposals for scientific society. Teach in 2nd year grad student course. Continue reviewing travel award proposals (there are 36 to do). Talk with undergrads in the lab about their projects.
Afternoon: Maybe spend two hours in the lab doing some research! Hopefully also work with lab tech learning new technique. Review more travel award proposals. Attend student seminar.
Friday-
Morning: I try to spend Friday mornings working at home, mostly writing and reviewing. I'll try to get a manuscript review done and finish the travel award reviewing.
Afternoon: Meet with people in the lab. Attend weekly meeting held by a Center I'm part of. Get back to work on that manuscript.
Okay, so maybe I don't have 36 travel award proposals to review every week. I may have more manuscripts or grant proposals to review. Or other committee meetings. Or travel. And I haven't detailed any of the constant interruptions that occur during each day. You can see it's difficult to carve out big blocks of time to spend in the lab during the week.
Such is life.
I wouldn't trade it for anything.
Monday, November 10, 2008
Undergraduates in research
There are a couple of interesting discussions going on over at Isis's bright, shiny new home regarding how undergraduates should approach PI's they're interested in working with, and how said PI's interact with such undergrads.
I am a big proponent of having undergrads working on research projects. I like working with undergrads in the lab. They tend to be enthusiastic and willing to learn. I have pretty much always had one or more in my lab right from the time I started at Big State U. A number of them have been extraordinarily productive. During the almost 12 years I've been here I've published seven papers with undergrads listed as authors. Six of those have an undergrad as first author. Another, with an undergrad first author, should be submitted in the next couple of weeks. Some readers of my blog may recall that I am also PI on an NSF REU Site grant that pays for a bunch of students to come do research in the department each Summer.
Based on my experiences with these students I offer up the following:
For undergraduate students-
For PI's-
Undergrads can rock in the lab. If given the chance.
I am a big proponent of having undergrads working on research projects. I like working with undergrads in the lab. They tend to be enthusiastic and willing to learn. I have pretty much always had one or more in my lab right from the time I started at Big State U. A number of them have been extraordinarily productive. During the almost 12 years I've been here I've published seven papers with undergrads listed as authors. Six of those have an undergrad as first author. Another, with an undergrad first author, should be submitted in the next couple of weeks. Some readers of my blog may recall that I am also PI on an NSF REU Site grant that pays for a bunch of students to come do research in the department each Summer.
Based on my experiences with these students I offer up the following:
For undergraduate students-
- Choose the PI's you're thinking of approaching carefully. Just because they're working on something you think is cool doesn't mean they're a good choice. Do they seem approachable? Have they had/do they have undergrads in their lab? Do they have a large lab, which would increase the chance that you wouldn't have much interaction with the PI? Check out their website for a start.
- Don't spam a bunch of PI's via email. Write each person you're interested in an individualized message. Indicate why you're interested specifically in them. Read up on the work they're doing. Every single undergrad who has ever worked with me scored a position because they had done their homework. And asked to talk to me.
- Be honest about why you want to try your hand at research. We do know that many premeds want the experience to increase their odds of getting into med school. Personally I have no problem with that - I'm in a college of medicine after all. Some PI's don't want premeds in their labs for a variety of reasons, some quite good - if you're premed you probably don't want to be in those labs. Besides, if we know what you want out of the experience we are in a better position to help you get it.
- A tepid reference letter for grad or med school from a "rockstar" professor won't help as much as an enthusiastic letter from a lesser known, more junior person. Tepid reference letters will often hurt your chances, not help.
- Be enthusiastic and be prepared to work hard. And remember that the PI has many responsibilities and may not always be available when you want/need them.
- Finally, keep in mind that many PI's at MRU's are not required to have undergrads in their labs. Paying tuition does not entitle you to a place in someones lab.
For PI's-
- Undergrads are a lot of work. Even the really good ones. But they are, IMHO, worth the effort. Heck, I've got seven, almost eight, publications as a result of working with undergrads. And they're fun.
- Don't take on an undergrad unless you're prepared to put in the work. You could assign them to a grad student or postdoc, but if you do, make sure you make some effort to stay involved in what they do. They came to work with you.
- Having a trial period in which an undergrad does scut work (washing dishes etc.) is fine. Just don't make it too long. Using an undergrad as free scut labor makes you a jerk. If you're paying them to do scut work and have no intention of getting them involved in research, make sure they understand that before they start.
- If an undergrad has earned authorship, give it to them. And put them in the right place in the authorship list. Bumping them out of a first authorship they've earned in order to give it to a grad student, postdoc or yourself makes you a serious ass wipe.
Undergrads can rock in the lab. If given the chance.
Saturday, November 08, 2008
Wednesday, November 05, 2008
Lab web site/wiki
In the comments to my earlier post on information management, Nat Blair wrote:
Here's another question though: what software do you use to store the tacit knowledge of the lab? Do you use a lab website to store some of that stuff? A lab wiki? Mostly just curious.
My answer at the time was that I wasn't using anything, but that I was considering implementing a wiki. There are a number of reasons why I hadn't implemented something. One was the draconian approach the administration here at Big State U. takes to any kind of web site created and maintained by faculty. A second was that implementing a wiki would require a substantial commitment of time on my part to get the software installed and running, plus an ongoing commitment to maintaining the software. I'm quite capable of doing this - I have a background in computational biology, although my lab is almost purely experiment-based now - but I'm not sure I can devote the time required. Nat's question did however spur me to spend some time looking for a simple solution. And I found one.
Google Sites.
There you can create a free web site with up to 100MB storage space. And, importantly, you can limit who can view and alter the site (e.g. upload or download stuff). Google (typically) provides a bunch of easy to use web-based tools for creating a site. So, with about two hours work I set up an intra-lab web site. I've uploaded many of our protocols and pdf's of some useful papers. There's a section including links to the web sites of the various vendors we commonly use. I've included a list gadget where any lab member can type in things that need to be ordered. Soon I hope to include a calendar gadget that will display everyone's schedules, including mine. It's a work in progress, but I think we're finally moving out of the dark ages with regard to lab information management.
Here's another question though: what software do you use to store the tacit knowledge of the lab? Do you use a lab website to store some of that stuff? A lab wiki? Mostly just curious.
My answer at the time was that I wasn't using anything, but that I was considering implementing a wiki. There are a number of reasons why I hadn't implemented something. One was the draconian approach the administration here at Big State U. takes to any kind of web site created and maintained by faculty. A second was that implementing a wiki would require a substantial commitment of time on my part to get the software installed and running, plus an ongoing commitment to maintaining the software. I'm quite capable of doing this - I have a background in computational biology, although my lab is almost purely experiment-based now - but I'm not sure I can devote the time required. Nat's question did however spur me to spend some time looking for a simple solution. And I found one.
Google Sites.
There you can create a free web site with up to 100MB storage space. And, importantly, you can limit who can view and alter the site (e.g. upload or download stuff). Google (typically) provides a bunch of easy to use web-based tools for creating a site. So, with about two hours work I set up an intra-lab web site. I've uploaded many of our protocols and pdf's of some useful papers. There's a section including links to the web sites of the various vendors we commonly use. I've included a list gadget where any lab member can type in things that need to be ordered. Soon I hope to include a calendar gadget that will display everyone's schedules, including mine. It's a work in progress, but I think we're finally moving out of the dark ages with regard to lab information management.
Monday, November 03, 2008
How the hell did that get published?????? [Updated]
[DISCLAIMER: This post was prompted, in part, by recent posts over at Isis's temple and DrugMonkey's cage. One should not take the following as a comment on the paper under discussion there - I have not read it and it's way outside my area of expertise, so I'm not making any judgments about it whatsoever.]
All too often I find myself reading a peer-reviewed paper and wondering how on earth it managed to get by the reviewers and editor, and end up being published. In a reputable journal. I know many of my colleagues have the same experience.
I'm not necessarily talking about disagreeing with the interpretation of data. Rather, it's a matter of poorly executed, incorrect or missing experiments, lack of suitable controls, incorrect statistical analyses, egregious lack of, or incorrect, citation of other published work in the area, etc. In other words, truly bad papers.
How do these get published? I don't know, but I would like to offer a couple of suggestions based on my experiences on the editorial board of a mid-level journal. First up, let me state that I am a strong believer in the peer review process. I am also well aware that it has it's flaws. Secondly, a little background...
Here's how things are supposed to work at the Journal of Doodlewidgets. When a manuscript is submitted, the authors must supply the names of four potential reviewers plus suggest a member of the editorial board as someone suitable for handling the review process. The Editor then assigns the manuscript to an editorial board member (EBM), who may or may not be the one suggested by the authors. The EBM is then supposed to read the manuscript and decide if it's good enough to send out for review or whether it should be rejected without full review. Those manuscripts deemed good enough are then sent out for review. When the reviews are returned to the journal, the EBM makes a final decision (accept, reject, major or minor revisions) and the authors are informed. Many journals in my field have similar processes for handling submissions.
Here's where I see the system breaking down too often: at the EBM level on two counts.
1) As noted above, at the Journal of Doodlewidgets the EBM is supposed to read the manuscript and decide if it's good enough to send out for review. In other words, the EBM is supposed to perform a preliminary review of the manuscript. I suspect (know) this doesn't happen in many cases. There are those who join editorial boards just for the extra line on their CV's, and who can't be bothered with applying the required effort. It's not clear how to handle these people. The obvious answer is to boot them, but that's not so easy if the offending party is a Big Cheese. Journals like to have Big Cheese's on their boards for the cachet. And don't want the negative publicity that might occur if they boot a Big Cheese...
2) The bigger problem lies in the choice of reviewers. It's all too easy to send the manuscript out to two (or more) of the suggested reviewers. Problem is, those people are likely good friends of the authors. We all play that game. We suggest people we know who we think will review our manuscripts fairly. Or, in the case of a (hopefully) minority of authors, automatically favorably. In some cases, authors suggest ex-co-authors or collaborators. The corresponding author of a manuscript I handled recently listed as a suggested reviewer someone they were co-authors with on a manuscript in press! Needless to say that's just not on.
So, it's possible that a given manuscript receives reviews that are more positive than they should be, and the EBM (who hasn't bothered reading the manuscript) simply accepts them at face value. Or the EBM is good friend of one of the authors and over-rides a more negative review. Or, perhaps the manuscript is somewhat outside the EBM's area of expertise, in which case they assume the reviews from the suggested reviewers are legit. In the end, a substandard manuscript can end up being accepted...
Note that this is the opposite of the usual reviewing issue (reviewers being too harsh) often discussed on the blogosphere. And I'm ignoring the issues of lazy reviewers - a good EBM should pick up on those - or unqualified reviewers - which is the EBM's fault for using them.
It's not clear how to deal with this. If all EBM's were conscientious it wouldn't be much of an issue. But how do you ensure that the editorial board is stocked with only the good? I suspect Editors have some idea of who's dead wood, but since there are no good metrics for measuring EBM performance...
EBM's could ignore the list of suggested reviewers, but then what's the point of making authors go through that process? And most authors are probably making legitimate suggestions. You could simply scrap the idea of having a list of suggestions, but then it becomes a real crapshoot if you have a lazy or somewhat unqualified EBM handling your manuscript.
I don't know the best way to fix this, or even if it's as big an issue as I perceive it to be, but I do have three suggestions. The first is for EBM's to use at most only one of the suggested reviewers. Yes, that means making some EBM's work harder, but it's not that much effort. I know because this is what I do. My second suggestion is to require authors to submit a list of people who should be excluded from reviewing due to various conflicts of interest. By this I mean the kind of thing the NSF requires from people applying for a grant. A list of all co-authors and collaborators over the last four years. Plus postdoc and PhD mentors. Okay, so that's a bunch of work for the authors, but you do want that manuscript reviewed fairly, right? And you do want to see fewer crap manuscripts accepted, right? If you keep a running list, it's not that much work. Alternatively, suggesting inappropriate reviewers could be made grounds for immediate rejection, but that can be a difficult call for even the most conscientious EBM.
And finally, the EBM's name should be included in the paper. That alone should improve matters.
And thus endeth yet another long post.
All too often I find myself reading a peer-reviewed paper and wondering how on earth it managed to get by the reviewers and editor, and end up being published. In a reputable journal. I know many of my colleagues have the same experience.
I'm not necessarily talking about disagreeing with the interpretation of data. Rather, it's a matter of poorly executed, incorrect or missing experiments, lack of suitable controls, incorrect statistical analyses, egregious lack of, or incorrect, citation of other published work in the area, etc. In other words, truly bad papers.
How do these get published? I don't know, but I would like to offer a couple of suggestions based on my experiences on the editorial board of a mid-level journal. First up, let me state that I am a strong believer in the peer review process. I am also well aware that it has it's flaws. Secondly, a little background...
Here's how things are supposed to work at the Journal of Doodlewidgets. When a manuscript is submitted, the authors must supply the names of four potential reviewers plus suggest a member of the editorial board as someone suitable for handling the review process. The Editor then assigns the manuscript to an editorial board member (EBM), who may or may not be the one suggested by the authors. The EBM is then supposed to read the manuscript and decide if it's good enough to send out for review or whether it should be rejected without full review. Those manuscripts deemed good enough are then sent out for review. When the reviews are returned to the journal, the EBM makes a final decision (accept, reject, major or minor revisions) and the authors are informed. Many journals in my field have similar processes for handling submissions.
Here's where I see the system breaking down too often: at the EBM level on two counts.
1) As noted above, at the Journal of Doodlewidgets the EBM is supposed to read the manuscript and decide if it's good enough to send out for review. In other words, the EBM is supposed to perform a preliminary review of the manuscript. I suspect (know) this doesn't happen in many cases. There are those who join editorial boards just for the extra line on their CV's, and who can't be bothered with applying the required effort. It's not clear how to handle these people. The obvious answer is to boot them, but that's not so easy if the offending party is a Big Cheese. Journals like to have Big Cheese's on their boards for the cachet. And don't want the negative publicity that might occur if they boot a Big Cheese...
2) The bigger problem lies in the choice of reviewers. It's all too easy to send the manuscript out to two (or more) of the suggested reviewers. Problem is, those people are likely good friends of the authors. We all play that game. We suggest people we know who we think will review our manuscripts fairly. Or, in the case of a (hopefully) minority of authors, automatically favorably. In some cases, authors suggest ex-co-authors or collaborators. The corresponding author of a manuscript I handled recently listed as a suggested reviewer someone they were co-authors with on a manuscript in press! Needless to say that's just not on.
So, it's possible that a given manuscript receives reviews that are more positive than they should be, and the EBM (who hasn't bothered reading the manuscript) simply accepts them at face value. Or the EBM is good friend of one of the authors and over-rides a more negative review. Or, perhaps the manuscript is somewhat outside the EBM's area of expertise, in which case they assume the reviews from the suggested reviewers are legit. In the end, a substandard manuscript can end up being accepted...
Note that this is the opposite of the usual reviewing issue (reviewers being too harsh) often discussed on the blogosphere. And I'm ignoring the issues of lazy reviewers - a good EBM should pick up on those - or unqualified reviewers - which is the EBM's fault for using them.
It's not clear how to deal with this. If all EBM's were conscientious it wouldn't be much of an issue. But how do you ensure that the editorial board is stocked with only the good? I suspect Editors have some idea of who's dead wood, but since there are no good metrics for measuring EBM performance...
EBM's could ignore the list of suggested reviewers, but then what's the point of making authors go through that process? And most authors are probably making legitimate suggestions. You could simply scrap the idea of having a list of suggestions, but then it becomes a real crapshoot if you have a lazy or somewhat unqualified EBM handling your manuscript.
I don't know the best way to fix this, or even if it's as big an issue as I perceive it to be, but I do have three suggestions. The first is for EBM's to use at most only one of the suggested reviewers. Yes, that means making some EBM's work harder, but it's not that much effort. I know because this is what I do. My second suggestion is to require authors to submit a list of people who should be excluded from reviewing due to various conflicts of interest. By this I mean the kind of thing the NSF requires from people applying for a grant. A list of all co-authors and collaborators over the last four years. Plus postdoc and PhD mentors. Okay, so that's a bunch of work for the authors, but you do want that manuscript reviewed fairly, right? And you do want to see fewer crap manuscripts accepted, right? If you keep a running list, it's not that much work. Alternatively, suggesting inappropriate reviewers could be made grounds for immediate rejection, but that can be a difficult call for even the most conscientious EBM.
And finally, the EBM's name should be included in the paper. That alone should improve matters.
And thus endeth yet another long post.
Subscribe to:
Posts (Atom)