Science Integrity

Some European institutions, according to a recently published Nature article, have started to rely on science integrity inspectors to review papers prior to submission for errors in manuscripts.¹ So far, this has not caught on in the United States. The question is, should it?

One of the institutes highlighted in the Nature article was the Fritz Lipman Institute (FLI) in Leipzig, Germany. Their director was accused of research misconduct and found guilty of not supervising his lab group properly. He ultimately resigned his position as director of the institute. This was the second case of alleged research misconduct within a year at this institute. The institute decided to strengthen its vetting process and hired a science-integrity inspector.

The FLI budgets about $55,000 per year to have a science integrity inspector look at their manuscripts. With 80-90 publications per year (according to Web of Science), this comes to about $650 per publication. This may not sound like much compared to the cost of the research that goes into a publication, but it becomes prohibitively expensive for large research universities that publish thousands of papers each year.

A science integrity inspector is a reactive way to deal with research misconduct. Research misconduct should not happen in the first place. Society expects scientists and researchers to behave in ethical ways. When scientists commit fraud, it has a disproportionate negative impact on the credibility of science.

A National Research Council report on Fostering Integrity in Research describes two theories for why researchers commit fraud.² One theory says that there will always be a small number of people who commit fraud and nothing can be done proactively to prevent fraud. Another theory allows for an environmental or cultural factor, for instance, the intense pressure researchers are under to find employment in academia and obtain funding year after year. This theory leaves room for effective interventions.

Universities in the U.S. and elsewhere are serious about scientific integrity. They have offices of compliance to investigate accusations of research misconduct and are proactive in training researchers in responsible conduct of research. This proactive stance on research misconduct is not by choice. Training in responsible conduct of research is a requirement of funding agencies.

Despite all these efforts, there is a perception “that a large and growing proportion of studies published across disciplines are unreliable due to the declining quality and integrity of research and publication practices.”³

How big a problem is it?

There have been numerous attempts to define the scale of unethical research behavior. A 2009 study by Daniele Fanelli⁴ showed that about 2% of scientists admitted to having fabricated, falsified or modified data or results at least once and up to 34% admitted other questionable research practices (QRPs). When asked about colleagues, scientists reported that they believed about 14% of their colleagues had falsified research results and up to 72% had committed other QRPs. Fanelli speculated that these were likely underestimates of the actual incidence because of the sensitivity of the issue. Almost ten years later, she concluded that “there is no evidence that scientific misconduct and QRPs have increased.”⁵

Serious errors can lead to a retraction of the paper. Retraction Watch, a blog that reports on retractions of scientific and engineering publications, has created a massive database of retracted papers. Brainard and You⁶ found that while “[t]he number of articles retracted by journals had increased 10-fold during the previous 10 years,” only about 4 out of 10,000 papers are being retracted. Not all retractions are due to fraud. According to Brainard and You, “nearly 40% of retraction notices did not mention fraud or other kinds of misconduct.”⁷

Lack of reproducibility is part of what falls into the 40% of retractions. It has become a serious issue over the past decade. Entire fields have had to face the reality that their research methods yielded irreproducible results. One of the first areas affected by this crisis was psychology. Many biomedical studies cannot be reproduced either. This was highlighted in the 2017 book by Richard Harris, “Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions.”⁸ Harris recounts Begley’s findings of barely being able to reproduce one in ten studies from academic labs while heading hematology and oncology research at Amgen and searching for promising ideas that might lead to new drugs.

How a hypercompetitive culture can lead to QPRs

Overstating claims to increase one’s impact is one outcome of the hypercompetitive culture. A recent study, published by Johnson et al. in the highly cited journal Proceedings of the National Academy of Science (PNAS), on “officer characteristics and racial disparities in fatal officer-involved shootings”⁹ falls into this category. The authors of this paper asserted in the Significance Statement that “White officers are not more likely to shoot minority civilians than non-White officers.”

The Significance Statement is written in a non-technical language and often used by journalists to convey to the public interesting scientific findings.

Racial disparity in shootings is of great public interest, and the statement was picked up by 33 news outlets and created 1970 tweets, according to Altmetric. To put the level of attention in context, this paper is currently in the top 5% of all research outputs scored by Altmetric. The only problem is
that the authors showed something much weaker, as was pointed out by Knox and Mummolo¹⁰, namely the likelihood of a victim of a shooting by a White vs. non-White officer being non-White.

The misleading statement could have been caught by a careful referee. But it wasn’t. And when a journal is vying for publicity, ethical standards may be further compromised by the journal itself, as happened in this case. When Knox and Mummolo pointed out the misleading statement to the editor, they had to fight for months with PNAS to publish a response that demonstrated that Johnson et al. could not have shown the claim in the Significance Statement with the data they had.¹¹

The hypercompetitive nature of research has also created an environment where researchers are tempted to “engage in careless or detrimental research practices.”¹² A recent example¹³ comes from a retraction by Frances Arnold, a Nobel Prize-winning scientist. She retracted a 2019 paper published in the journal Science after she found out that the results were not reproducible. She tweeted that she wasn’t as careful as she should have been since she “was a bit busy when this was submitted.”

Detecting research fraud

Plagiarism and image manipulation are common problems. Both can be detected with available software, and, increasingly, publishers rely on software to screen manuscripts for those problems prior to publication.

Advances in data mining have led to new tools to detect research misconduct. For instance, Markowitz and Hancock found that “[f]raudulent papers were written with significantly higher levels of linguistic obfuscation, including lower readability and higher rates of jargon than unretracted and nonfraudulent papers.”¹⁴

However, data mining approaches that can scan a very large number of publications in very little time also come with a downside. These methods produce false positives, that is, they will flag publications that are not fraudulent. When a large number of publications are scanned, this may result in a significant number of publications that are falsely accused of QRPs. Even false accusations have a chilling effect on authors, are time-consuming to respond to, and may threaten the authors’ reputation.

Can science integrity inspectors prevent research misconduct?

If someone wants to cheat to get ahead, they will find ways to do so, even if a science integrity inspector will review the manuscript. Instead of reactive measures, like science integrity inspectors, we believe that measures that establish a culture that encourages ethical behavior will ultimately be more successful. This includes the establishment of social norms of honest behavior and responsibility to both the scientific field and the public. Training on responsible conduct of research will need to be part of this, as well as ways to report research misconduct without fearing repercussions. Practices that reduce the temptation to cheat, such as pre-submission of experimental protocols and requirements to share raw data, need to become the standard if we want to ensure that the public trusts science.

Claudia Neuhauser, Ph.D., is the Associate Vice President of Research and Technology Transfer and Professor of Mathematics at the University of Houston.

Brian Herman, Ph.D., is a Professor in the Department of Biomedical Engineering and former Vice President for Research at the University of Minnesota and University of Texas Health, San Antonio.

---

1. Abbott, Alison. “The science institutions hiring integrity inspectors to vet their papers.” (2019): 430-433.
2. National Academies of Sciences, Engineering, and Medicine. 2017. Fostering Integrity in Research. Washington, DC: The National Academies Press.
3. Fanelli, Daniele. “Opinion: Is science really facing a reproducibility crisis, and do we need it to?.” Proceedings of the National Academy of Sciences 115.11 (2018): 2628-2631.
4. Fanelli, Daniele. “How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data.” PloS one 4.5 (2009): e5738.
5. Fanelli, Daniele. “Opinion: Is science really facing a reproducibility crisis, and do we need it to?.” Proceedings of the National Academy of Sciences 115.11 (2018): 2628-2631.
6. Brainard, Jeffrey, and Jia You. “What a massive database of retracted papers reveals about science publishing’s ‘death penalty’.” Science 25.1 (2018): 1-5.
7. Ibid.
8. Harris, Richard. Rigor mortis: how sloppy science creates worthless cures, crushes hope, and wastes billions. Basic Books, 2017.
9. Johnson, David J., et al. “Officer characteristics and racial disparities in fatal officer-involved shootings.” Proceedings of the National Academy of Sciences 116.32 (2019): 15877-15882. (Retracted by the authors on July 28, 2020)
10. Knox, Dean, and Jonathan Mummolo. “Making inferences about racial disparities in police violence.” Proceedings of the National Academy of Sciences 117.3 (2020): 1261-1262.
11. Knox, D. and J. Mummolo. It took us months to contest a flawed study on police bias. Here’s why that’s dangerous. The Washington Post. January 28, 2020.
12. Ibid. p. 102
13. Nobel Prize-winning scientist Frances Arnold retracts paper. BBC News. January 3, 2020.
14. Markowitz, David M., and Jeffrey T. Hancock. “Linguistic obfuscation in fraudulent science.” Journal of Language and Social Psychology 35.4 (2016): 435-445.