Can we trust clinical trials from China?
Can we trust clinical trials from China?
(Or anywhere else?)
China has supplanted the U.S. as the largest producer of scientific articles. More and more Chinese clinical trials are being published in Western medical journals, including elite publications like the New England Journal of Medicine, JAMA, and The Lancet. The trend can only be expected to grow.
Many of these trials report new findings that could significantly impact the practice of medicine in the U.S. and worldwide.
Yet fraud is believed to be widespread in Chinese research, and can be difficult to detect. How much should Western physicians trust the results of clinical trials conducted in China?
The Prevalence of Fraud in Medical Research
Fraud is widespread in biomedical research and publishing generally. When Retraction Watch started as a side project blog by Ivan Oransky and Adam Marcus in 2010, they wondered if they would have enough material. They needn’t have worried. Over its first decade and a half RW built an ever-expanding database of more than 50,000 retracted journal articles requiring a team of staffers to curate, supported by funding from the MacArthur Foundation and other philanthropies. The database is now co-administered by Crossref and continues to grow.
Oransky and Marcus believe the approximately 5,000 retracted journal articles each year represent only a tiny fraction of the total number of fraudulent papers: they estimate more than 100,000 per year should be retracted. (About 1.5 million biomedical research studies are published each year. Most published research findings are false or irrelevant, but not fraudulent.)
Journals “let foxes guard the henhouse” and “are invariably more interested in protecting their reputations and the reputations of their authors than in correcting the record,” Oransky and Marcus wrote in a 2023 op-ed. A U.S. federal agency, the Office of Research Integrity, is responsible for investigating allegations of research misconduct in federally funded research, but rarely does, says Oransky. Eager to protect their own reputations, universities have been notoriously slow to act in cases of suspected misconduct, either.
High-volume paper-mill journals pumping out trash papers are by far the largest source of problematic research. But increasingly sophisticated scrutiny by amateur sleuths has also regularly implicated elite institutions and journals.
Multiple papers published by authors from Harvard-affiliated Dana Farber Cancer Institute have been retracted for image manipulation, duplication or other serious problems, according to Sholto David at For Better Science. Marc Tessier-Lavigne, past president of Stanford University, resigned after student journalist Theo Baker’s reporting led to an independent commission’s conclusion that work the president supervised was substandard with “multiple problems,” leading him to retract multiple papers from Cell and Science and correct two in Nature.
Medical academics are under immense pressure to publish papers, especially in higher-impact journals. If they succeed, they’ll gain research funding, prestige, and power. If they don’t, they’ll lose their jobs or (gulp) be forced to do more clinical work.
Researchers know any serious forensic evaluation of their work is extremely unlikely; they and their university can intimidate accusers with actual or threatened lawsuits; and even if caught red-handed, they probably won’t face serious repercussions.
Dutch researchers (in science and the humanities) admitted as much in a large survey: one in twelve (8%) of almost 7000 respondents admitted fabricating results within the past year. More than half said they regularly hide flaws in their research design or engage in other dubious tactics.
And that’s in the Western world, where the idea of academic integrity was invented and (despite prominent deviations) cultural norms of fair play and neutral rules still tend to govern most collaborative activity in the public sphere.
Now, let’s head to China.
China Has Incentivized Research Misconduct—
While Western researchers have strong temptations to engage in misconduct, the incentives in China have been outright egregious.
From the 1990s through 2020, Chinese researchers were paid large bonuses for publishing in medical journals, as part of a formal government policy to accelerate universities’ research output.
According to a 2017 article in the MIT Technology Review, “scientists who publish in the top Western journals can earn in excess of $100,000 per paper.” At that time, the average salary for a university professor was $8,600 per year.
This would be the equivalent of an American primary investigator receiving a bonus of $2.3 million for publishing a single paper.
Even if shared among multiple collaborators, that’s a big haul. (Publishing in lower-impact journals netted much smaller bonuses, in the low thousands—although researchers could publish more articles more easily in those journals.)
China banned payment for publication in 2020, stating the policy was creating “perverse incentives,” according to an article in Nature.
—With Predictable Results
There is ample evidence that fraud and other research misconduct are even more widespread in China than in Western nations:
As of 2017, the retraction rate for Chinese papers was three times what should be expected: they had 8% of the published papers but 24% of the retractions.
By 2024, almost half of the 50,000 retracted articles in Crossref’s and Retraction Watch’s database were from China.
Three-quarters of the ~14,000 papers retracted from English language journals in 2023 involved a Chinese co-author, according to Nature.
Fake peer review, plagiarism and paid publication in paper-mill journals occur on a massive scale.
Surveyed Chinese scientists and engineers said they believed that 40% biomedical papers are tainted by research misconduct. (Other Chinese surveys put the figure at 20%.)
Reform is Hindered by Cultural Factors
The Chinese government has initiated multiple reform measures in an attempt to de-incentivize or punish research misconduct.
But what Westerners consider “misconduct” is often considered acceptable within some segments of Chinese society:
Surveyed Chinese doctoral students report a high tolerance for research misconduct: in 2022, more than one quarter said falsifying some data was okay.
Plagiarizing without attribution is considered acceptable in textbooks and other reference sources in China.
Chinese students are more likely to be dismissed from U.S. universities for violating academic integrity policies (e.g., plagiarism), possibly due in part to these differences, according to a white paper by WholeRen Education.
While such beliefs and values persist (which also exist to a lesser degree in Western countries), it’s hard to imagine significant improvement inside China in what Westerners would consider academic integrity.
Western Journals Can’t Be Trusted to Vet Chinese Papers
Medical journals in the U.S. are unlikely to vet Chinese papers with the degree of skepticism they deserve.
For one, they can’t. The first layer of protection against fabricated data comes from the nearby peers and colleagues at a researcher’s own institution or competing institutions. This process can’t be trusted to operate in China.
Second, the Chinese government ultimately controls research production there. If achieving a particular research result were desirable, there is no question they would be able and willing to produce that result.
It seems unlikely that medical journals would perform forensic evaluations of data underlying Chinese research, or that they would be able to detect fabricated data if they did. (They have already shown themselves to be largely unable or unwilling to do so for research with a lower baseline rate of fraud.)
Even if journals could detect fraud consistently, they have reason not to look too closely. Medical journals need content, and China can provide it in abundance.
Just this year, there have been multiple impressively positive randomized trials emerging from China in the critical care literature alone. They may all be completely authentic—or not. It’s impossible to say. But it will be interesting to see how many more emerge, at what rate, and how credulously they are accepted for publication by major Western medical journals.
Conclusion
Academic misconduct occurs everywhere, but is common and widely tolerated in China.
The Chinese government incentivized academic fraud by paying bonuses for publication in Western medical journals for almost 30 years (supposedly stopping in 2020).
The results are seen today in huge numbers of retractions of Chinese research papers, far exceeding expected rates.
This is unfortunate and unfair for the many researchers in China who seek only to produce work of high scientific integrity in the pursuit of truth.
Medical journals have become dependent on Chinese research output, but are neither able nor willing to adequately vet it for integrity.
This publication is unable to adequately vet Chinese clinical research, either. Because important and true research findings are likely to be reported from China, it’s not appropriate or fair to ignore Chinese papers. Neither is it wise to ignore the increased likelihood of misconduct, though.
Going forward, articles from China will be reviewed on PulmCCM but marked with an asterisk(*) to allow readers to discount the information, or not, according to their own judgment.
Your opinion on this challenging issue is welcome—please comment below.
References and Links
The Truth about China’s Cash-for-Publication Policy, MIT Technology Review, July 2017
“Why Fake Research Is Rampant in China,” The Economist, February 2024
The End of Publish or Perish? China’s New Policy on Research Evaluation, Max Planck Institute
Five ways China must cultivate research integrity, Nature, November 2019
Nearly half of US Chinese student dismissals due to “academic dishonesty”, The PIE, September 2023
Dana-Farberications at Harvard University, For Better Science, January 2024
How Much Scientific Research Is Actually Fraudulent? Reason, July 2021
Thank you for this post. I have been making the same point in lectures specifically dealing with postoperative delirum research, where there has been a suspicious pattern in data. Studies on the effect of intraoperative BIS (pEEG) monitoring has shown a tendency to be favourable only in China, or more specifically only when the patients were randomized in China. I felt a bit uncomfortable pointing this out, but was confirmed recently in this paper - a meta-regression, finding highly skewed data on delirium prevention both by pEEG and Dexmedetomidine, when the data was from China. I would say that this is the best example of a smoking gun as of yet. The problem is that our current concept of meta analysis is simply not valid, if it includes fraudulent data which propably means all of them. https://www.bjanaesthesia.org.uk/article/S0007-0912(24)00349-0/fulltext
I agree with the article and have been suspicious regarding studies out of China for a long time. My only objection to your analysis is that this is not a "cultural norm" of the Chinese people as evidence by the fact that most Chinese scientists and physicians are not accepting this (whether in Taiwan, other SE Asian countries, or those locally who want to maintain integrity). It is, however, the norm of the communist and Marxist principles regardless of the field or country overtaken by this ideology.