Would you participate in a "virtual" NSF review panel in Second Life?
As John Bohannon describes, NSF has been running them for two years, saving $10,000 per panel in the process ("Meeting for Peer Review at a Resort That's Virtually Free"):
Since March 2009, six grant-review panels have convened on NSF's island in Second Life, known as IISLand. “Realworld panelists are provided with some resources,” says Bainbridge. “So it was felt appropriate to provide them with the cost of a decent set of virtual clothes.” Once the scientists had created avatars, they each received 1000 Linden dollars—which cost NSF $4—to shop in Second Life's virtual stores. (They also received a $240 honorarium of real money per day.)
Yeah, I've got to think that's going to skew the pool of reviewers. I don't see myself sitting around for two days on a virtual panel. For many good scientists, I have to think this would impede their ability to contribute productively.
The sheer amount of manpower involved in grant review is amazing:
Over the past year, more than 19,000 scientists traveled to NSF headquarters in Arlington, Virginia, to take part in traditional review panels. Most of them worked for two solid days, huddled in groups of six to 10, carefully reading, discussing, and scoring dozens of research proposals competing for over $6 billion in grants. It's the scientific equivalent of jury duty. NSF covers expenses, but the small honorarium—typically $500—hardly covers a scientist's time, especially considering the days lost to travel. But how else can NSF evaluate the merits of all those proposals?
And that's just for the panels who evaluate applications and compare the external reviews. It doesn't count the work of the thousands of external reviewers, who are unpaid and unrewarded.
I think it would work just as well to find a few objective metrics, retrodict grant success based on those metrics for the past five years, and then use a discriminant function to give out the money. Anything that cuts down on reviews and bureaucracy would net increase productivity. Sure there would be people gaming the system, but there are with the current system!
