Peer Review or Vetting?

Also at ethericstudies.org/peer-review-or-vetting/
These practices are recommendations provided under the
Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License
Also, See Failure to Replicate Fallacy, Survival-Related Media Review, and Rating and Peer Reviewed Online Journals

Abstract

A technical report intended for publication is considered more credible if it has been subjected to peer review. peer review is considered and intended results are compared to actual results. An alternative approach known as vetting is discussed and a recommendation is made for how vetting might be used.

Introduction

These comments are written from the point of view that there is an Academic-Layperson Partition in the paranormalist community which impairs cooperation between parapsychologists and experiencers. The effect is that people who are trained in the scientific method tend to distrust the often more pertinent understanding of paranormal phenomena held by experiencers and practitioners.

A second assumption is that the lay members of the paranormalist community tend to assume parapsychologists honestly seek to further understanding of Psi Field and survival-related phenomena. But, in fact, some parapsychological literature is designed to show that the phenomena are normal mistaken as paranormal, fraud or delusion. And that such debunking literature is often written in a way that laypeople do not clearly understand.

Terms

Paranormalist is defined here as people who are more than a little interested in paranormal phenomena, or who are studying or practicing some technique related to the paranormal.

The paranormalist community consists of people in some way interested in paranormal phenomena.

Experiencers are those who have witnessed an apparent objective paranormal event or who have had an apparent paranormal personal experience.

Practitioners are those who are able to produce objective and subjective paranormal phenomena.

Paranormalist interests include the study of:

Phenomena

Psi phenomena (telepathy, remote viewing, telekinesis)
Healing intention (biofield healing, distant healing, healing prayer)
Hauntings investigation, transcommunication (mediumship, ITC, channeling)
Survival (the etheric, personality, cosmology, possibly reincarnation)

Experiences

Near-Death
Out-of-Body Experiences (Soul Travel, Astral Projection, Sometimes simple disassociation)
Reincarnation (past life regression, sense of previous lives)
The grief of losing a loved one
Fear of the unknown (fear of dying, fear of demonic forces, fear of becoming earthbound)

From mainstream society, paranormalists are conditioned to trust scientists, almost without question. Within the paranormalist community, parapsychology represents the science subgroup seeking to further understanding of paranormal phenomena and experiences. They are the academically trained people to whom the rest of the community turns for answers.

Problem Areas

There is an Academic-Layperson Partition in the Paranormalist Community which probably naturally evolved because of a difference in education. The mostly Ph.D. parapsychologists are accustomed to a professional and university culture which is mostly not part of the layperson’s experience. The effect is that those who are best prepared to cooperate to study these phenomena are not effectively communicating with those who, as experiencers and practitioners, most understand their practical nature.

There are a number of important consequences of this partition:

Collaboration

Information flow is mostly from academic to layperson by way of publication and conferences. There is very little flow of information in the other direction.
Without an open exchange of information from practitioners and experiencers, researchers have been shown to incorrectly assume understanding.
Lacking authentic information about the target phenomenon, research protocols are often based on wrong assumptions.
The parapsychological community exhibits a sort of Wizard Complex in which there is an assumption of knowledge that is not, in fact, knowledge (Omniscient Science).

Communication

Parapsychologists have been shown to have three primary research objectives: Anomalistic Psychology (Physical Hypothesis) (1); Exceptional Experiences Psychology (Super-Psi Hypothesis) (2); and, Survival (Survival Hypothesis) (3).
A high percentage of material published by the parapsychological groups is intended to prove the Anomalistic Psychology or Exceptional Experiences Psychology point of view. This, without clearly stating as much in the article.
The combination of ill-informed protocol design and deceptive research objectives in respected journals has made those journals mostly irrelevant to the larger community.

Ethics

The Academic-Layperson Partition has produced a culture within the parapsychological community which has apparently made it permissible for researchers to deceive laypeople.
There are instances of researcher mistreatment of layperson research subjects. The standard of ethical conduct required by universities is effectively ignored by parapsychologists.
There is no outcry from parapsychologists when one of their members is unethical. Instead, the parapsychological community effectively circles the wagons in their support.
The parapsychological community furthers layperson abuse and deceptive communication by creating opportunities for the offending researchers to tout their misleading discoveries.

Literature Visibility

There are many factors influencing the effect of peer review practices and in truth, probably none are malicious or intended to suppress frontier subjects. However, while all of these may be unintended consequences, they are also virtually all controlled by the mainstream academic culture.

The resulting document becomes part of the body of literature which is accessed by other researchers, thus multiplying positive and negative aspects of the published document.
Publication editors are able to select comments from the public, and thereby control the apparent acceptability of the article.

For frontier subjects which may include emergent science, the concerns also include:

Mainstream authority is virtually always the only source for both author and peers. Consequently, the public has been taught to respect academically credentialed scientists with little reservation and peer-reviewed articles are seen to represent the truth about the subject.
In practice, literature produced by laypeople studying frontier subjects is simply ignored by academics because of the lack of academic standing of the authors and laypeople’s inability to be published in academically respected journals.
Frontier subjects become represented by the mainstream academic community, which by contrast, biases public perception.
Public funding for research and education in frontier subjects depends on the perception of the subject fostered by mainstream academia and is consequently mostly unavailable to the often better-informed laypeople.
It can be said that academic careers flourish or fade depending on how often their writing is cited. The same is true of rankings in Internet search results. Citing is part of the academic culture. Citing is not as common amongst laypeople. Consequently, lay literature tends to fade from lack of notice.

Review and Rating

A distinction is made in this essay between academic peers and subject-matter specialists. Academic peers are those who, from an academic point of view, are treated as having equal education and possibly equal ability. Subject-matter specialists are people who have established themselves in their field of study or practice as people who have practical understanding of the subject.

Peer Review

In the context of technical articles and research reports proposed for publication, peer review is a process by which a document is evaluated by people academically trained in the appropriate field of study and who are seen as being sufficiently qualified to judge the quality of the article. The review is intended to be conducted during preparation for publication.

The current practice is to deliberately keep the reviewer’s identity and qualifications secret. Reviewer comments to authors are also kept secret. As a practical consequence of secret peer review, the validity of parapsychological research reports is necessarily in question. This problem is exasperated when obviously biased articles, and reports about poorly designed studies, are published in supposed peer-reviewed journals.

A common complaint amongst paranormalists is that, without oversight, a good old boy culture appears to have developed in which review of club values (procedure, formatting, author credentials) may be accepted as peer-review, while subject matter content may not be appraised.

Some practitioners have noted that they are unaware of parapsychologists who are knowledgeable about the practitioner’s study and who might have been a peer reviewer for a journal. While one practitioner may not be aware of qualified parapsychologists, the complaint is so common that practitioners have little choice but to think related articles are only reviewed for compliance with editor objectives and construction but not for technical reasonableness.

Vetting

As it is intended here, vetting is the examination of published material by subject-matter specialists to determine the sensibility of the material. In the paranormalist community, subject-matter specialists tend to be practitioners and laypeople studying the phenomena in their natural circumstances. (In this case, experiencers are not considered subject-matter specialists unless they have also conducted studies and are practitioners or have a direct understanding of their work.)

Because of the Academic-layperson Partition, vetting is more likely to occur after publication. Consequently, the results of vetting tend to be in the form of reviews posted on social media or on personal websites.

Vetting is intended to warn future readers about what they should expect from the media. It should always be done with the additional intention of providing feedback to the authors. Thus, vetting should be conducted as a positive process with positive results in mind.

Negative reviews put the entire paranormalist community in a poor light and increase distrust. With this in mind, reviews should be couched in terms of what has and what can be learned from the experience. Reviewers should be mindful that all is not known about these phenomena. It is risky to be very dogmatic about what is right.

In vetting, it is important that more than one person is part of the process. As recommended in the Survival-Related Media Review and Rating (Draft) Best Practice, (4) an average rating representing a consensus of the reviewers should accompany reference to the reviewed document.

Three example reports of vetting are:

Failure to Replicate Fallacy (5)

Debunking Survival Under Cover of False Academic Authority (6)

Failure to Replicate ITC (7)

Consideration for Vetting Media

As with peer review, vetting is ideally conducted prior to publication. With that in mind, here are a few of the characteristics a reviewer may wish to consider (also see Survival-Related Media Review and Rating (4)):

Clearly stated reason – Reason for the publication is clearly evident. This is not the research question. It is the reason the research question is being asked. What is the author trying to accomplish? Is it to further understand or to prove something?
Relevance – Does the media indicate its scope? For instance, should the media be directed toward a human behavior-related audience such as the field of psychology or sociology? If it is a study of group interaction incidentally conducted in a typically paranormal situation such as a hauntings investigation, it probably has nothing to do with the paranormal aspect and should be marked accordingly. The next question would have to be, “if so, why is it being published in a parapsychological journal?”
Furthering Understanding – It should be clearly stated if the media is simply replicating old studies. If the authors have incorporated new theories or used a special technique for replication, it would be helpful if that is mentioned early on. A witness report is not a research report. If a peer-reviewed journal publishes a witness report as science, look for ulterior motives such as debunking.
Collaboration – Have the authors included lay-literature in their preparation? Have they made an effort to have practitioners and experiencers review the protocol and considered their suggestions? Such information should be clearly noted in the introduction of the media. Be aware that, in some cases, collaboration is with people who are handy, but only peripherally part of the sub-community. The author may have unwittingly contacted the least qualified, self-proclaimed expert. If so, provide helpful suggestions.
Theory – An important way to further understanding is to at least attempt to incorporate research findings into the model that represents the author’s assumptions. Have contending theories been considered? If so, why were they rejected? It is not necessary to address all contending theories. Perhaps the main three: Physical Hypothesis, Super-Psi Hypothesis and Survival Hypothesis.
Ethics – Ethical considerations include treatment of human test subjects and fairness in criticism. At a minimum, the standard in The Belmont Report (8) should be followed for human test subjects. If in the media, did the authors fairly comment or did they assume knowledge they might not have? Will the media further the community or will it diminish the community?
Clarity of Communication – Is the media composed so that a person who is not trained in the subject at a Ph.D.-level should be able to understand the message? This would be true of at least the introduction (the abstract) and the conclusions. An overreliance on multi-syllable words which would be unfamiliar to someone in a different field of study should be noted as a negative. If a statistical analysis was conducted, are the results clearly stated rather than requiring an in-depth understanding of statistical notation? Is the intended meaning communicated in a way that will be useful to the average paranormalist?
Availability – The media need not be free to the public, however, if it is not, are there any parts of it for which the public might have a need to know? This is especially important for media that draws a presumably learned conclusion about phenomena important to others studying the subject, or that finds fault with others. If the media is behind a paywall, any related media review should list that as a decided negative. (4)

Note that laypeople subject matter specialists are not expected to comment on the application of the scientific method. Protocol design is dependent on factors that are unique to the authors understanding and purpose. Reviewers should probably limit comments to how reasonably the protocol treated techniques and current practices.

A final editorial comment by the reviewer may be useful if the conclusions seem to stray too far from current understanding without explanation as to why. Editorial comments are risky, however, in that they can easily become the unnecessary promotion of the viewer’s personal favorite theory.

Vetting is Collaboration

The objective of any publication is to communicate something the authors believe others should or would like to know. If the author is deceptive about the intent or vague about the media’s content, the resulting communication might be better considered propaganda intended to debunk or subtly change experiencer’s opinion.

Peer review is part of a closed system embedded in the academic side of the Academic-Layperson Partition. It is not visible to laypeople, and therefore, the actual meaning and value of peer-reviewed literature are not evident to laypeople.

In contrast, vetting of media by members of the intended audience is a more visible way of helping experiencers understand the media and authors understand how well they accomplished their objective.

References

APStaff. “What is Anomalistic Psychology?” Goldsmiths, University of London. 2015. gold.ac.uk/apru/what/.
Simmonds-Moore, Christine. “What is Exceptional Psychology?” Journal of Parapsychology (#76 supplement, 54-57, 2012).
Butler, Tom. “Trans-survival Hypothesis.” Etheric Studies. 2015. ethericstudies.org/trans-survival-hypothesis/.
Draft Best Practice. “Survival-Related Media Review and Rating.” Etheric Studies. 2018. ethericstudies.org/practice-srm-media-review/
Butler, Tom. “Failure to Replicate Fallacy.” Etheric Studies. 2018. ethericstudies.org/failure-to-replicate-fallacy/
Butler, Tom. “Debunking Survival Under Cover of False Academic Authority.” Etheric Studies. 2014. ethericstudies.org/scientist-attack-medium/
Butler, Tom. “Failure to Replicate ITC.” Etheric Studies. 2010. ethericstudies.org/failure-to-replicate-itc/
“The Belmont Report: Office of the Secretary, Ethical Principles and Guidelines for the Protection of Human Subjects of Research.” The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research.1979. hhs.gov/ohrp/humansubjects/guidance/belmont.html.