Is peer review a reliable guide to reliability?

[eight bits]

I've posted this essay (a long read, but currently more-or-less viral in some sectors of scholarship) in Phil & Psy, because its author is a psychologist: https://www.adammastroianni.com/ . Although many of the examples are from "hard" science and the history of science, most of the problems discussed in the essay are the same as those found in many disciplines, including the humanities and social sciences.

Here on the wild, wild web many of the skeptical persuasion come across as having a faith-based confidence in the importance of peer-review. I've occasionally posted in reply that what peer review mainly certifies is that a paper (book, etc.) satisfies the publisher's editorial standards (which vary widely) and that the editor has some assurance that the paper will be of some interest to the target readership. There is little or no warranty that the work is reliable in any strong sense.

The essay criticizes peer review far more deeply than that, even going so far as hinting that the peer review "stamp of approval" may be outrightly deceptive regarding the merits of a work. Ouch.

Without further ado, here's the essay:

https://experimentalhistory.substack.com/p/the-rise-and-fall-of-peer-review

(WARNING: It's not peer-reviewed .)

[Ell]

1 hour ago, eight bits said:

many of the skeptical persuasion come across as having a faith-based confidence in the importance of peer-review

That is a contradictio in terminis.

 

I suspect that it is mostly psychopaths and narcissists who insist on peer reviewed papers. They are unable to judge truth or falsehood themselves.

[+and-then]

5 minutes ago, Ell said:

That is a contradictio in terminis.

 

I suspect that it is mostly psychopaths and narcissists who insist on peer reviewed papers. They are unable to judge truth or falsehood themselves.

I have no idea if that's commonly accepted but I do know that the peer-review practice is used with physicians in many cases.  Not every patient but a random sampling to hold docs accountable and to function as an additional source of continuing education.  I think when applied to scholarly work it may be less credible, depending on the system that's used to examine the efforts of others.  These days money has polluted almost everything.

[Ell]

Most physicians are idiots.

 

For example, in an anamnese for cardiology they ask, as they did me: "did dead any relatives have heart problems?" Insinuating that their problems and your disease are likely congenital.

 

What they do not ask me is: "were those dead people treated by physicians or cardiologists?", nor "did those dead people have a chronic herpes virus infection or another chronic viral infection?", nor "did those dead people have cancer or brain haemorrhages before they were treated for cardiac problems?", nor "did those dead people have cancer or brain haemorrhages after they were treated for cardiac problems?"

[+Hammerclaw]

Does anyone here consider themselves the scholastic equal to the peer group? If not, then what merit your dissent?

Edited January 29, 2023 by Hammerclaw

[pallidin]

2 hours ago, eight bits said:

... I've occasionally posted in reply that what peer review mainly certifies is that a paper (book, etc.) satisfies the publisher's editorial standards (which vary widely) and that the editor has some assurance that the paper will be of some interest to the target readership. There is little or no warranty that the work is reliable in any strong sense...

You are soooooo wrong and blatantly defamatory to the established norms of educational reviews.

You make it sound as if the Editor functions as GOD and above the peer group.

They call it "peer review" because certified like-minded multi-individual community members considers the merits of the material... NOT THE EDITOR!!!! DUH!!!!!!!

Hence the term "peer"!!!!!!

For whatever reason you are grossly twisting things here, but whatever. Bye.

[Alchopwn]

5 hours ago, eight bits said:

I've posted this essay (a long read, but currently more-or-less viral in some sectors of scholarship) in Phil & Psy, because its author is a psychologist: https://www.adammastroianni.com/ . Although many of the examples are from "hard" science and the history of science, most of the problems discussed in the essay are the same as those found in many disciplines, including the humanities and social sciences.

(WARNING: It's not peer-reviewed .)

It may well be that Artificial Intelligence becomes our peer review method in the future.

[Tiggs]

8 hours ago, eight bits said:

I've posted this essay (a long read, but currently more-or-less viral in some sectors of scholarship) in Phil & Psy, because its author is a psychologist: https://www.adammastroianni.com/ . Although many of the examples are from "hard" science and the history of science, most of the problems discussed in the essay are the same as those found in many disciplines, including the humanities and social sciences.

Here on the wild, wild web many of the skeptical persuasion come across as having a faith-based confidence in the importance of peer-review. I've occasionally posted in reply that what peer review mainly certifies is that a paper (book, etc.) satisfies the publisher's editorial standards (which vary widely) and that the editor has some assurance that the paper will be of some interest to the target readership. There is little or no warranty that the work is reliable in any strong sense.

The essay criticizes peer review far more deeply than that, even going so far as hinting that the peer review "stamp of approval" may be outrightly deceptive regarding the merits of a work. Ouch.

Without further ado, here's the essay:

https://experimentalhistory.substack.com/p/the-rise-and-fall-of-peer-review

(WARNING: It's not peer-reviewed .)

Sounds roughly right.

I'm not sure that absolute guarantees of reliability exist, outside of tautologies. Fairly sure that even some of the Laws of Physics come with Terms & Conditions (*valid at this point in Time, for the known universe only. All constants subject to change, without prior notification.).

IMO, some of the journals offering peer review probably aren't worth the paper they're written on.

But in general — peer review from a respected journal > non-peer review, in reliability terms.

[pallidin]

18 minutes ago, Tiggs said:

...But in general — peer review from a respected journal > non-peer review, in reliability terms.

Exactly. So, his argument fails by virtue of contradicting well-established norms.

Destructive perception/expression of a time-honored and proven practice is not only not good, it's disturbing.

Edited January 29, 2023 by pallidin

[eight bits]

8 hours ago, and-then said:

I have no idea if that's commonly accepted but I do know that the peer-review practice is used with physicians in many cases.  Not every patient but a random sampling to hold docs accountable and to function as an additional source of continuing education.  I think when applied to scholarly work it may be less credible, depending on the system that's used to examine the efforts of others.  These days money has polluted almost everything.

To be clear, "peer review" as used in the essay refers to pre-publication vetting of scholarly writing. There are plenty of other institutional arrangements in the learned professions that might be decribed as "review of a professional's work by other professionals." Medicine has its hospital tissue committees, institutional review boards, licensing agencies, professional conduct committees, and so forth.

You might even define a vocation, in part, as a "profession" by the existence of such institutions. The real world effectiveness of such institutions would make a fine topic, but it is not the current topic. As is implied by the historical survey in the focus essay, such professional institutions existed in the absence of peer review in literature. They really are, then, separate things.

The alternative to peer review in medical journals is NOT surgeons who are moody loners with knives and saws.

-

7 hours ago, Hammerclaw said:

Does anyone here consider themselves the scholastic equal to the peer group? If not, then what merit your dissent?

Not that I know of, and even within whatever field, typically even the most accomplished scholars do not consider themselves equal to the universe of their peers. That is simply not the issue here (or anywhere else).

A typical peer review is conducted by a handful of invited and then self-selected (volunteer) scholars. It is not, does not purport to be, nor can it possibly be a useful random sample of the universe of interested and qualified scholars (n = 2, 3 or 4?).

As I've posted over the years and the focus essay alludes to, the real evaluation of scientific merit occurs after publication, when the entire community has the opportunity to examine the goods and determine their usefulness. The peer review that concerns us here is a step before publication or to prevent publication (or as frequently happens, to determine where something is published, not whether it is published at all).

-

6 hours ago, pallidin said:

You make it sound as if the Editor functions as GOD and above the peer group.

The prerogatives of the editor-in-chief vary among journals and other publishers. The final power resides in the publisher, but effectively that power is often very heavily delegated. Whether an EIC views themeselves as a god is between them and their mental health care provider.

-

2 hours ago, Alchopwn said:

It may well be that Artificial Intelligence becomes our peer review method in the future.

I think it is inevitable that AI will play a role in the dissemination of scholarly writing. What that role is may be difficult to predict. Perhaps it isn't so much to restrict what is published as to recommend to working scholars what they need to read, filtered from an ocean of content available through the Web.

In the old days (as today will be in a very few years), scholars would subscribe to services like "current abstracts" where the titles, authorship, and  the short (usually 50-300 words) self-descriptions of papers appearing in a range of journals would be gathered, sorted, indexed by keyword, etc. The scholar would then skim through this list of potential reading, and pick and choose what they found promising to read in full. Even as we speak, Google Scholar pretty much does that indexing chore on demand and for free.

With AI in the loop, I might not even have to frame a query. "Brief me, Alexa." Even today, she knows what I want and what I need. Tomorrow? She will shape me into the man she knows I can become .

-

59 minutes ago, Tiggs said:

But in general — peer review from a respected journal > non-peer review, in reliability terms.

Yes, if something in a given field appears in Nature or Science, then it's probably worth reading by everyone working in that field. But then, if an outsider wanted some reassurance about that, or to apply the principle to less well-known or more specialized outlets, then they'd do well to turn to impact factors (measures of how much of the content of a journal is then cited in other scholarly works). Impact factors are post-publication measures.

@pallidin brought up the divine status of editors. It should be noted that there is a step prior to peer-review which in principle occurs everywhere, but at high profile journals like Nature and Science it is a substantial bar for a would-be author to clear. That is, the editorial decision whether or not to circulate a submission for peer review. If the submissions editor (probably not the EIC at the more prominent journals, but more likely somebody much lower in the pecking order) says no, then barring divine intervention, the submission is rejected without further review.

Which as the focus essay points out, really means that the rejected paper probably gets submitted elsewhere, and in all likelihood (providing only that it is not written in crayon on butcher paper) will appear somewhere proudly bearing the imprimatur of having successfully been peer-reviewed.

-

44 minutes ago, pallidin said:

Destructive perception/expression of a time-honored and proven practice is not only not good, it's disturbing.

As the focus essay states, it isn't all that time-honored, and whether or not it is proven is what we are here to discuss.

Edited January 29, 2023 by eight bits

[Davros of Skaro]

Peer Review is not bullet proof. Even accredited institutions can make mistakes. Not to mention PRs that are self published (tinfoil hat stuff).

[+Hammerclaw]

It would be helpful to the OPs argument if he were to post examples of Peer Review gone wrong.

[pallidin]

6 hours ago, eight bits said:

 

@pallidin 

As the focus essay states, it isn't all that time-honored, and whether or not it is proven is what we are here to discuss.

You are confusing scientific publishing standards and the peer review process with the idea that the submitted paper has "all the proof"

It has never been that way.

During peer review, what is suggested instead is that the paper is deemed acceptable for publication, and that its content may be of further professional interest. It may or may not have any embedded proofs.

So, peer review does not "proof" anything. It's a multi-individual stamp of potential (but not guaranteed) publication acceptance within that field and within that field's accepted standards of research and publication.

It really is that simple.

 

 

Edited January 29, 2023 by pallidin

[eight bits]

3 hours ago, Hammerclaw said:

It would be helpful to the OPs argument if he were to post examples of Peer Review gone wrong.

He did discuss some of the experimental work (submitting papers salted with deliberate errors, for example). Also, Google is your friend. Peer review gone wrong is a fertile search string.

4 hours ago, Davros of Skaro said:

Peer Review is not bullet proof. Even accredited institutions can make mistakes. Not to mention PRs that are self published (tinfoil hat stuff).

But "self published" covers a wide range of material these days. That includes preliminary reports of material that is about to be submitted to peer-reviewed outlets. In really fast moving fields (most recently COVID vaccine and other countermeasure work), the breaking news gets informally shared and then shows up later in journals (unless it's obsolete by then or known to be a dead end).

It's hard to evaluate that breaking news channel of information. Ivermectin for COVID was a bad idea; off-the-shelf steroids to prevent heart disease when kids got COVID was a great idea. The dissemination of the latter was heavily dependent on personal-trust networks, initially involving person-to-person chats among physicians in New York City hospitals. Never mind peer review, there wasn't even a "clinical trial" - "Hey, Joe, my 14 year old patient was dying yesterday and isn't dying anymore" (which is called anecdotal evidence, just in case somebody thought that was an oxymoron).

It's a hard problem. It wouldn't be worth discussing if it was easy or already solved.

ETA: @pallidin You seem to claim to know me well. Have we ever met? In any case, I believe you misread my comment. You described peer-review as a "proven practice." I pointed out that that was what was in controversy in the focus article and this thread.

I said nothing about "the idea that the submitted paper has 'all the proof,'" nor whether submitted papers include proofs (some do, others don't if you mean formal proofs). We seem to agree that surviving peer review means the paper conformed to standards of its publisher, whose business being what it is means that the publisher predicts that there will be some reader interest in the paper. I suspect we disagree whether those standards are unform from one publisher to the next within a field.

 

Edited January 29, 2023 by eight bits

[pallidin]

15 minutes ago, eight bits said:

...It's a hard problem. It wouldn't be worth discussing if it was easy or already solved.

It HAS already been solved, long before you or I were even born!!!

There is no problem at all.

Peer review is a necessary component of submission dynamics and has nothing to do with the paper being 100% correct.

That part is left for the final researchers and their own peer reviewed submissions with the "final conclusion"

 

Edited January 29, 2023 by pallidin

[pallidin]

"Proven practice" means that the approach is proven, not necessarily the data.

Used in Law, Medicine and countless other professions.

[Grim Reaper 6]

16 hours ago, eight bits said:

I've posted this essay (a long read, but currently more-or-less viral in some sectors of scholarship) in Phil & Psy, because its author is a psychologist: https://www.adammastroianni.com/ . Although many of the examples are from "hard" science and the history of science, most of the problems discussed in the essay are the same as those found in many disciplines, including the humanities and social sciences.

Here on the wild, wild web many of the skeptical persuasion come across as having a faith-based confidence in the importance of peer-review. I've occasionally posted in reply that what peer review mainly certifies is that a paper (book, etc.) satisfies the publisher's editorial standards (which vary widely) and that the editor has some assurance that the paper will be of some interest to the target readership. There is little or no warranty that the work is reliable in any strong sense.

The essay criticizes peer review far more deeply than that, even going so far as hinting that the peer review "stamp of approval" may be outrightly deceptive regarding the merits of a work. Ouch.

Without further ado, here's the essay:

https://experimentalhistory.substack.com/p/the-rise-and-fall-of-peer-review

(WARNING: It's not peer-reviewed .)

A penny for Peer Review!

[eight bits]

9 minutes ago, pallidin said:

Peer review is a necessary component of submission dynamics and has nothing to do with the paper being 100% correct.

As the focus article points out, the practice was rare before the 1960's. And yet there was a scientific literature back then. It follows that the practice is not necessary. It has some attractive features and some less attractive features, hence the interest in the focus article among scholars, and a purpose for this thread in our own cozy corner of the web.

9 minutes ago, pallidin said:

"Proven practice" means that the approach is proven, not necessarily the data.

And so I responded to your earlier remark. If what you wish to say is that the author of the focus article is mistaken, then great. You can make your point without groudless speculation about how old I am, whether I understand common English expressions, how many years if any I served on the editorial board of a peer-reviewed journal, or how many reviews I have received and written. (I am on the record at UM as having survived one peer review for a conference paper last year; anything else would require off-site research on your part.)

[pallidin]

I was once a well-traveled District Manager, and my esteemed mentor said to me "Do not present a complaint or problem without a suggestive solution because it makes you sound like a whiner"

And that was only one of his awesome admonitions to me.

Seriously, those were his words to me as I rose through the ranks.

-----------------

All I am hearing here are complaints.

Curiously, also about what I view as a non-issue.

 

[Grim Reaper 6]

47 minutes ago, eight bits said:

As the focus article points out, the practice was rare before the 1960's. And yet there was a scientific literature back then. It follows that the practice is not necessary. It has some attractive features and some less attractive features, hence the interest in the focus article among scholars, and a purpose for this thread in our own cozy corner of the web.

I believe that reviewed work certainly has its pros and cons. While there many publishers who peer review work it appears to me that some publishers are head and shoulders above others. Eightbits it seems like the Publisher "Nature" is one of those publishers. Is this accurate@eight bits?

29 minutes ago, pallidin said:

I was once a well-traveled District Manager, and my esteemed mentor said to me "Do not present a complaint or problem without a suggestive solution because it makes you sound like a whiner"

And that was only one of his awesome admonitions to me.

Seriously, those were his words to me as I rose through the ranks.

-----------------

All I am hearing here are complaints.

Curiously, also about what I view as a non-issue.

 

1 hour ago, pallidin said:

"Proven practice" means that the approach is proven, not necessarily the data.

Used in Law, Medicine and countless other professions.

47 minutes ago, eight bits said:

And so I responded to your earlier remark. If what you wish to say is that the author of the focus article is mistaken, then great. You can make your point without groudless speculation about how old I am, whether I understand common English expressions, how many years if any I served on the editorial board of a peer-reviewed journal, or how many reviews I have received and written. (I am on the record at UM as having survived one peer review for a conference paper last year; anything else would require off-site research on your part.)

 

Edited January 29, 2023 by Grim Reaper 6

[eight bits]

54 minutes ago, Grim Reaper 6 said:

I believe that reviewed work certainly has its pros and cons. While there many publishers who peer review work it appears to me that some publishers are head and shoulders above others. Eightbits it seems like the Publisher "Nature" is one of those publishers. Is this accurate@eight bits?

Yes, Nature is universally regarded as a high-quality, high-prestige journal. 

-

1 hour ago, pallidin said:

All I am hearing here are complaints.

Sometimes the first step in solving a problem or improving an existing way of doing things is to realize that there is a problem or that the existing way has serious shortcomings. A reasonable second step would be to specify the problem or inventory the shortcomings. A reasonable third step is to issue a call to a variety of interested and qualified people to work on remedying or improving the situation. All of which the focus author has done.

The author also took an active step on his own. As he reported in his essay, he self-published a paper and arranged for the right people to see it and read it (= wrote it in such an engaging way that when they saw it, they wanted to read it).

Again, there's no big deal if you disagree with the focus author,

1 hour ago, pallidin said:

Curiously, also about what I view as a non-issue.

But it is clear that given his belief that there is an issue, he has proceeded in a rational and responsible way to mitigate the harm which he believes the current way of doing business inflicts on scholars.

[pallidin]

I'm done with these incessant, blatantly ridiculous attacks on the values and immense importance of "peer reviews"

Bye, bye!

[eight bits]

For the benefit of those who haven't played this particular game, here is some documentary footage of what happens when the first-round of reviews arrive. Caution: adult language from both sides of the Atlantic.

 

[eight bits]

Here's a follow-up article by the OP's focus author.

I see this issue as parallel to when automobiles were new. Without rails to connect stations, how will so-called  "drivers" know where to go and how to get there? Won't they get lost? What if two automobiles come to an intersection at the same time? How will they know who goes first? Won't we have to place signals everywhere to tell them? What happens if the automobile runs out of fuel in a lonely place? For that matter, where's this "fuel" supposed to come from?

Those, and many other issues as well, were problems once upon a time. But accommodation of technological improvement happened. In part, that was because the previously dominant "last mile" transport technology, however charming horses are, involved shoveling a fair amount of droppings. That, too, was a problem, even though horse carts were legendary for their "self driving" feature (the horse learned the delivery route and knew not to trample pedestrians, etc.) which only now can automotive tech almost match.

https://slate.com/human-interest/2023/01/peer-review-critique-scientists-controversy.html

Oh, as this ad from Mercedes-Benz dramatizes, a certain amount of effort will be required to adjust to new technological realities. As the ad also illustrates, we humans have an awesome track record in rising to those challenges.

 

[OpenMindedSceptic]

No. Peer reviewed doesn't give reliability. Ask any amateur Egyptoligist to share their peer reviewed timeline of the pharaohs. Ask almost any biologist which pharma companies fund their labs. Ask any physicist how electrons really work...