Discuss Peer Review

[jgoodguy]

Saw this post

Whether something is peer-reviewed or not is really irrelevant though. That is really just an indirect appeal to authority, The problem has to do with the quality of scholarship, not whether a work is peer-reviewed and the two are not necessarily the same thing.

I really do not understand your argument. The fact of peer review shows certification of the article suggesting a quality of scholarship not in non peer reviewed articles. In short anyone can write a non peer reviewed article and publish it. Maybe it is quality or maybe not.

How to recognize peer-reviewed journals

Peer-reviewed (refereed or scholarly) journals - Articles are written by experts and are reviewed by several other experts in the field before the article is published in the journal in order to insure the article's quality. (The article is more likely to be scientifically valid, reach reasonable conclusions, etc.)​

What Is A Peer-Reviewed Article? - Evaluating Information Sources

What Is Peer Review?
In academic publishing, the goal of peer review is to assess the qualityof articles submitted for publication in a scholarly journal. Before an article is deemed appropriate to be published in a peer-reviewed journal, it must undergo the following process:
​

• The author of the article must submit it to the journal editor who forwards the article to experts in the field. Because the reviewers specialize in the same scholarly area as the author, they are considered the author's peers (hence "peer review").

• These impartial reviewers are charged with carefully evaluating the quality of the submitted manuscript.

• The peer reviewers check the manuscript for accuracy and assess the validity of the research methodology and procedures.

• If appropriate, they suggest revisions. If they find the article lacking in scholarly validity and rigor, they reject it.

· Because a peer-reviewed journal will not publish articles that fail to meet the standards established for a given discipline, peer-reviewed articles that are accepted for publication exemplify the best research practices in a field.
​

I am not sure what an "an indirect appeal to authority" is but consensus as in a peer reviewed article is not an appeal to attorney.
Scientific Consensus: NOT An Argument from Authority

In contrast, the scientific consensus is not an argument from authority. There are a couple of ways to think about this. The most basic and concise is that the scientific consensus is not based on an individual's or small group's credibility.​

In short peer reviewed has better credibility than non peer reviewed.

 

[E_just_E]

As someone who has been in the Academia and went through the peer review wringer a few times (and once very recently), I agree that peer reviewed articles are better quality than not-peer reviewed, in general, thus the former being published in journals and the later in magazines, blogs, or become white papers.

The above is accurate but there are nuances:

• Because the reviewers specialize in the same scholarly area as the author, they are considered the author's peers (hence "peer review").

• These impartial reviewers are charged with carefully evaluating the quality of the submitted manuscript.

Exactly because the reviewers specialize in the same area as the submitting author, and because government funding for research is very competitive, in addition to peers, they are competitors. Some of them more than others. Plus, because in some fields there are only a handful of people who go to the same conferences etc., there are social forces and subgroups and friendships and the opposite of friendships, creating conflict of interest in peer review. In order to minimize that journal editors allow authors to name names that should be excluded from being reviewers because of conflict. So, in some cases, the reviewers ended up being someone's buddies and co-grantees on occasion. But in general crappy work does not go through no matter who the reviewer is, just borderline crappy work; whereas in magazines and blogs the sky is the limit.

As far as books go there is no peer review in general, so quality there varies. And the bigger the publisher it is, the bigger the money that needs to be made, thus publicity and potential revenues trump quality everywhere.

 

[DaveBrt]

Peer reviewing has taken hard hits recently as peer reviewed articles have been proven to be full of errors and even fraud. It seems that peer reviewing is not what it used to be. In fact, in STEM it is understood that the original experiments will not be replicated -- though most of us would expect that.

A major problem is finding a peer reviewer -- one who is an expert in the same field as the author, but who has no ax to grind and will review fairly. I was twice told my book could not be "peer reviewed" because no one else knew the field.

 

[chucksr]

• The author of the article must submit it to the journal editor who forwards the article to experts in the field. Because the reviewers specialize in the same scholarly area as the author, they are considered the author's peers (hence "peer review").

• These impartial reviewers are charged with carefully evaluating the quality of the submitted manuscript.

There's the rub with peer review--reviewers are not always impartial and the author, I'm sure, cherry picks his "peers" before forwarding the manuscript to make sure there is some like mindedness in the process. I'm sure the good and the bad with peer review depends on the topic, the science, the field that is being written. I can see where the process of peer review could retard rather than advance knowledge in some, maybe most fields, creating "cookie cutter" scholars. Imagine what might happen if an author researched and discovered that "Global Warming" was not a man made peril! Do we imagine that this conclusion, no matter how well researched and valid scientifically, would be treated by "peers" in the environmental science field would be objective about that conclusion?
While peer review might help guarantee proper scientific methods were applied, you can bet your boots that peer review also serves to perpetuate long held beliefs and common understandings in a whole range of fields--history, sociology, public affairs, environmental science, educational methods, public education all come to mind as fields of study that might be infected with a degree of "don't rock the boat" and a rigid presumption of group think.

 

[James Brenner]

It's been quite some time since I was in academia - and my memories could be faulty - but, during the time I was there, the mantra was "publish or perish". As a result, those seeking tenure tended to publish very scholarly, peer-reviewed pieces in academic journals that I felt were meant primarily to impress the tenure committee, rather than inform any broader audience. Topics were often obscure, theoretical, trendy, or esoteric and received little or no attention in wider circles. Those historians who strayed into mass market publishing became essentially second-class citizens among their colleagues. My point is this: if, for whatever reason, research fails to contribute to a greater understanding or awareness, it's doing itself and the discipline a disservice.

Peer review does often guarantee quality scholarship, but publishing an article that few will ever read, does not seem like a good use of talent, time, or resources.

 

[Malingerer]

There's the rub with peer review--reviewers are not always impartial and the author, I'm sure, cherry picks his "peers" before forwarding the manuscript to make sure there is some like mindedness in the process. I'm sure the good and the bad with peer review depends on the topic, the science, the field that is being written. I can see where the process of peer review could retard rather than advance knowledge in some, maybe most fields, creating "cookie cutter" scholars. Imagine what might happen if an author researched and discovered that "Global Warming" was not a man made peril! Do we imagine that this conclusion, no matter how well researched and valid scientifically, would be treated by "peers" in the environmental science field would be objective about that conclusion?
While peer review might help guarantee proper scientific methods were applied, you can bet your boots that peer review also serves to perpetuate long held beliefs and common understandings in a whole range of fields--history, sociology, public affairs, environmental science, educational methods, public education all come to mind as fields of study that might be infected with a degree of "don't rock the boat" and a rigid presumption of group think.

First, you don't get to pick the "peers" doing the review - they are completely anonymous and unknown to each other. These are volunteer positions that fall to experts in a particular field. I know this firsthand because as a plant ecologist, I've been on both sides of the process. Secondly, in the natural sciences, new ideas, especially ones that run counter to accepted thinking are very much encouraged - although we treat virtually every assertion with some skepticism - as long as the evidence supports the claims. Maybe in other scholarly fields this kind of conservatism exists but I don't work in those fields and wouldn't care to speculate about something I know nothing about.

 

[Josh The Lighthouse Guy]

Do we imagine that this conclusion, no matter how well researched and valid scientifically, would be treated by "peers" in the environmental science field would be objective about that conclusion?
While peer review might help guarantee proper scientific methods were applied, you can bet your boots that peer review also serves to perpetuate long held beliefs and common understandings in a whole range of fields--history, sociology, public affairs, environmental science, educational methods, public education all come to mind as fields of study that might be infected with a degree of "don't rock the boat" and a rigid presumption of group think.

Long-established theory needs to be overturned with great care. Look at anti-vax movement stemming from a single flawed paper that has since been discredited. If one were to write a paper attempting to prove Flat Earth the uphill battle would and should be exceptionally steep no matter how good the author's research seems.

History isn't science. It's much more plausible to write a new look at a battle or general, especially with extensive new primary source research. But as the saying goes: "Extraordinary claims require extraordinary evidence."

 

[E_just_E]

History isn't science.

Beg to differ. History is a Social Science and should have as much scrutiny as Life and Natural Sciences. Every hypothesis and conclusion should be supported by data. Any assumptions made when there is no data should be denoted as such. It is ok to say that "we don't know" instead of making stuff up.

On the other hand, not everything that calls itself "History" is History. That "History" is not Science, is just a "Story", loosely based on reality, no matter how nice it flows or sounds.

 

[Josh The Lighthouse Guy]

I would also like to point out not all peer review is academic. I recently proofread a book by a fellow lighthouse historian. Neither of us is an academic (I work for a non-profit, not a university; I'm not aware of any lighthouse academics - maybe there are some maritime historians with a specialty in the subject?) However, there are a number of competent professionals and knowledgeable amateurs. I was able to offer corrections to misstatements and question certain claims and assertions that a proofreader not familiar with the subject would probably overlook.

I would imagine non-academics like @Eric Wittenberg get their books peer reviewed by other Civil War historians, academics or otherwise.

 

[E_just_E]

I was able to offer corrections to misstatements and question certain claims and assertions that a proofreader not familiar with the subject would probably overlook.

That's not proofreading, it is line editing Distinctions here.

 

[Eric Wittenberg]

I would imagine non-academics like @Eric Wittenberg get their books peer reviewed by other Civil War historians, academics or otherwise.

Absolutely. I have a network of friends--some academics, some not--who review pretty much everything that I write for me. They know that I will not be offended by their criticism and that I have no ego about such things. They catch mistakes. They make suggestions for improvement. And my work is better for their efforts. I am grateful to each of them for their contributions and for what they do to improve my work.

Because I am grateful to them for what they do, I try to return the favor whenever I can.

 

[AshleyMel]

This is an interesting thread. Hubby is back in college after years and years and had to "relearn" how to write his research papers due to sourcing methods now and peer review methods. He was amazed he just couldn't pick up an encyclopedia or reference someone book or article like "in the old days". Hubby is no lightweight. He's a smart fella and he finds some of this peer reviewed stuff problematic but he's managing just fine. I am not smart like you fellas, I appreciate your perspectives, I will share it with him.