The New Atlantis has this article up: Rethinking Peer Review: How the Internet is Changing Science Journals
In recent times, the term “peer reviewed” has come to serve as shorthand for “quality.” To say that an article appeared in a peer-reviewed scientific journal is to claim a kind of professional approbation; to say that a study hasn’t been peer reviewed is tantamount to calling it disreputable. Up to a point, this is reasonable. Reviewers and editors serve as gatekeepers in scientific publishing; they eliminate the most uninteresting or least worthy articles, saving the research community time and money.
But peer review is not simply synonymous with quality. Many landmark scientific papers (like that of Watson and Crick, published just five decades ago) were never subjected to peer review, and as David Shatz has pointed out, “many heavily cited papers, including some describing work which won a Nobel Prize, were originally rejected by peer review.”
This is something I'd never actually considered, though I suppose after reading various histories of science, one would more or less get it by osmosis that much earlier work was never formally peer-reviewed"; it got published if editors liked it, and then it was either accepted or rejected. The peer-review process was distributed.
The Cochrane Collaboration, an international healthcare analysis group based in the U.K., published a report in 2003 concluding that there is “little empirical evidence to support the use of editorial peer review as a mechanism to ensure quality of biomedical research, despite its widespread use and costs.” The Royal Society has also studied the effects of peer review. As the chairman of the investigating committee told a British newspaper in 2003, “We are all aware that some referees’ reports are not worth the paper they are written on. It’s also hard for a journal editor when reports come back that are contradictory, and it’s often down to a question of a value judgment whether something is published or not.”
That's somewhat true. I've had reviews that seemed more or less, well, in a word, dumb. That is, brief dismissals of nearly the entire thing, after which it goes on to be published in that same journal. And there's certain politics involved: no matter how dumb you think the comments are, you still need to politely and formally address all of them in your resubmittal.
. . .peer review has been criticized for being used by the scientific establishment “to prevent unorthodox ideas, methods, and views, regardless of their merit, from being made public” and for its secretiveness and anonymity.
Eh, this doesn't seem to be much of a problem, except for those papers that aren't too far out of the mainstream. Virtually anything can get published somewhere; not getting it in a peer-reviewed journal means it doesn't have the seal of authority (which the article goes some way to call into question) but all sorts of crap ends up being put out into the public domain one way or another.
The article also touches on a couple of online-only "journals" (if you can call them that) that are experimenting with other methods of review, such as posting initial drafts and letting anyone review it, let the manuscript get updated depending on comments, etc. It makes the review-and-revise process fairly dynamic instead of static.
But this goes back to this post that referenced an Althouse post on the place of law journal articles in an Internet-driven world. I wrote:
Journals are, I think, the high point of the profession and mark the standard of dialog that we use to hash out the big theoretical and methodological issues of the day. They have to be carefully reasoned and presented with adequate supporting data in order to provide a proper basis for discussion. Having the work set in stone, as it were, gives the whole process a baseline to work from. Creating a set piece of work that can be reviewed and presented at a single time and place fulfills that.
I still don't know how one might reference a constantly-evolving piece of work. References are shorthand for arguments we don't necessarily want to spend a lot of time on, but what if a particular reference keeps changing? People change their minds and their thinking evolves, but one can always reference later works to reflect that. Depends on how this online constant-review process works; it seems to me there needs to be some point at which one can set what one thinks in stone so that it can be a set point for discussion.