My favorite section of this post is the middle section in which you discussed your thoughts about “fraud fraud.” And my reaction was: “Gasp! and OMG!”
I ended up skimming through your AI Claude quotes (I bet most of your readers did too). It presented a few interesting points, but what would have happened if you the writer had done a typical Glenn Reynolds pithy and critical summary instead. (I guess we are all just experimenting with our chatbots these days!)
My two cents! Good job!
And, is there any way to get rid of those revolting Crohn’s disease ads on Instapundit?
“…typical Glenn Reynolds pithy and critical summary”
I originally became a regular reader of Instapundit over two decades ago because of the “typical Glenn Reynolds pithy and critical summary.” And that is the reason I continue to be an avid and continuous reader of Instapundit and a paid subscriber of this Substack. I also appreciate the biting humor, often intertwined within the pithy and critical summary, and the pertinent comments!
One way to dramatically reduce scientific fraud is to eliminate government funding of science. Ideally, a proper government exists only to protect individual rights. It does not "do" things, rather, it acts only as a retaliatory force. We are nowhere near that and I'm aware that the likelihood of getting the government out of research funding (or legislating the economy) is virtually nil. Nonetheless, reducing it as much as possible would shift focus and emphasis to the most important or promising research. Fewer government-funded research projects would also make them easier to scrutinize.
My worry is that because of predictable mission creep, instead of merely being banned on Facebook or losing credentials, scientists will face imprisonment for conducting unpopular studies or reaching unpopular conclusions. People might then ask, "Who are the real criminals?"
Excellent presentation. My favorite of the suggestions is making the raw data, including software modeling, public. To take one example, all of anthropogenic warming climate models would be immediately cratered by this. All the worst offenders, like Professor Hockey Stick, have refused for decades to release raw data and the software. If the methodology is secret, it can’t be reproduced or studied, and is therefore not science.
Failure to deliver factual information is more a problem in morals than in science or politics. Institutions that fail morally must be rebuilt with people who believe in liberty informed by morals grounded in reverence for life. Corrupt people in law enforcement is no proof against corrupt people in science.
Peer reviewers should demand that each submitted paper include a section explaining how the paper's results could in fact be wrong. Any paper not including this section is rejected automatically. Among other benefits, it may cause the original researcher to be more circumspect about their work.
would ending government funding be a solution? This assumes the private sector could take it over and it also assumes the private sector would insist on better results because they would be investing dollars that would want some sort of return on investment.
I posted my comment at about the same time you posted yours and only saw it after-the-fact. Private sector funding would not eliminate fraud, but it greatly reduces the number of people who pay for fraudulent results.
"That would slow down the connection between research and public policy, but would that really be such a bad thing?"
Not only would it not be a bad thing, it would be a good thing. There is no reason to think that public policy--and especially federal policy--should be treated differently than writing new laws. Going slow is mostly better than going fast.
Then again, it would take new public policy to accomplish that, and that's where I contradict myself.
I am not a lawyer, but scientific (research) fraud is already a criminal offense. Specifically, if you engage in research fraud during an FDA regulated clinical trial and you are found out, the penalty is a felony conviction (making false statements - lying - to the government) and jail time. See, for example, the case of Dr. Robert Fiddes in the late 1990's. Unfortunately, it is pretty easy to cover up research fraud.
"And – and this was suggested by a commenter to an earlier blog post – not relying on scientific research for public policy purposes until it has been successfully replicated by someone else is not a bad idea." Sounds like something I'd say, and maybe I did! But the point is broader than that. The broader point is based on the observation that the vast majority of published science has no impact at all (not on public policy, not on textbook writers, not on reporters in the media, and not on other scientists). One approach to the replication crisis--which is popular--is to make sure that virtually all published findings (no matter how obscure and irrelevant) will replicate, even at great expense to the taxpayer. Another approach is to instead focus limited resources on published findings that gain traction, not just in terms of public policy but also in terms of media attention, attention from other scientists, attention from textbook writers, etc. NSF or NIH could fund proposals that make the case that the to-be-replicated results are gaining traction, so precious taxpayer dollars should be devoted to independently replicating those findings in a large-N study. That's a good reason to spend taxpayer dollars. Spending that money to instead ensure that everything replicates, not so much. Moreover, even reforms that seem free are more costly than they appear to be (e.g., in terms of a scientist's limited time). Just fund proposed replications of published findings that are gaining traction. It would solve so much of the problem. Unfortunately, NSF and NIH are both reluctant to do that. They'd rather fund "transformative" (often non-replicable) experiments. Fine, but put aside 5-10% of the federal science budget to replicate published results that matter. Why cheat if NSF is going to quickly fund an independent replication of the findings that are (momentarily) making you famous?
For all govt grants, require data to be archived in the money receiving institution And a copy in the US Library of Congress, And a copy with the org that publishes the study.
Set aside some 50% of grant money for replication studies, to decrease as the most cited studies are replicated or their non-replication is published.
Bounty rewards for those who find fraud in others’ work.
John Ioannidis has written extensively on this. One of his recommendations is to use P values of 0.01 instead of 0.05. That would definitely make P-hacking more challenging. Another reasonable response would be to not approve a drug or intervention until a 3d party meta-analysis shows benefit. We also need to figure out some way to remove big pharma funding of studies. Their incentives to cheat are just too great when they provide the funding for a study that if the results are positive will result in that company making billions.
Most journals require an analysis of possible study flaws in the discussion section, so that option is already in play.
Criminalizing science "fraud" would hamstring science, especially if the government was in charge of the analysis as well as the penalty. We've learned during the pandemic what Moderna and Pfizer will do to foist their vaccines on the public without appropriate study and analysis. Their ability to do that went straight to the bottom line. Many companies would not hesitate to invoke criminal intent on a rival company to protect market share and earnings.
Bioscience and medicine must more effectively monitor itself. Current practices at the NIH should be modified. Making one person or a small group responsible for the distribution of billions in grant funds to favored researchers is a practice that is an invitation to misuse. It most recently led to decisions that were illegal and resulted in the development of a virus that resulted in our most recent global pandemic. Clearly changes need to be made, and we in medicine must do it ourselves.
Glenn, you should continue to write on this, it's a great topic.
My favorite section of this post is the middle section in which you discussed your thoughts about “fraud fraud.” And my reaction was: “Gasp! and OMG!”
I ended up skimming through your AI Claude quotes (I bet most of your readers did too). It presented a few interesting points, but what would have happened if you the writer had done a typical Glenn Reynolds pithy and critical summary instead. (I guess we are all just experimenting with our chatbots these days!)
My two cents! Good job!
And, is there any way to get rid of those revolting Crohn’s disease ads on Instapundit?
Roger von Oech
Paid Subscriber (from day 1)
“…typical Glenn Reynolds pithy and critical summary”
I originally became a regular reader of Instapundit over two decades ago because of the “typical Glenn Reynolds pithy and critical summary.” And that is the reason I continue to be an avid and continuous reader of Instapundit and a paid subscriber of this Substack. I also appreciate the biting humor, often intertwined within the pithy and critical summary, and the pertinent comments!
One way to dramatically reduce scientific fraud is to eliminate government funding of science. Ideally, a proper government exists only to protect individual rights. It does not "do" things, rather, it acts only as a retaliatory force. We are nowhere near that and I'm aware that the likelihood of getting the government out of research funding (or legislating the economy) is virtually nil. Nonetheless, reducing it as much as possible would shift focus and emphasis to the most important or promising research. Fewer government-funded research projects would also make them easier to scrutinize.
My worry is that because of predictable mission creep, instead of merely being banned on Facebook or losing credentials, scientists will face imprisonment for conducting unpopular studies or reaching unpopular conclusions. People might then ask, "Who are the real criminals?"
Excellent presentation. My favorite of the suggestions is making the raw data, including software modeling, public. To take one example, all of anthropogenic warming climate models would be immediately cratered by this. All the worst offenders, like Professor Hockey Stick, have refused for decades to release raw data and the software. If the methodology is secret, it can’t be reproduced or studied, and is therefore not science.
And then there is climate science destroying entire economies
Failure to deliver factual information is more a problem in morals than in science or politics. Institutions that fail morally must be rebuilt with people who believe in liberty informed by morals grounded in reverence for life. Corrupt people in law enforcement is no proof against corrupt people in science.
Peer reviewers should demand that each submitted paper include a section explaining how the paper's results could in fact be wrong. Any paper not including this section is rejected automatically. Among other benefits, it may cause the original researcher to be more circumspect about their work.
would ending government funding be a solution? This assumes the private sector could take it over and it also assumes the private sector would insist on better results because they would be investing dollars that would want some sort of return on investment.
I posted my comment at about the same time you posted yours and only saw it after-the-fact. Private sector funding would not eliminate fraud, but it greatly reduces the number of people who pay for fraudulent results.
require a result to reproduced by someone not party to the initial research before publication.
"That would slow down the connection between research and public policy, but would that really be such a bad thing?"
Not only would it not be a bad thing, it would be a good thing. There is no reason to think that public policy--and especially federal policy--should be treated differently than writing new laws. Going slow is mostly better than going fast.
Then again, it would take new public policy to accomplish that, and that's where I contradict myself.
I am not a lawyer, but scientific (research) fraud is already a criminal offense. Specifically, if you engage in research fraud during an FDA regulated clinical trial and you are found out, the penalty is a felony conviction (making false statements - lying - to the government) and jail time. See, for example, the case of Dr. Robert Fiddes in the late 1990's. Unfortunately, it is pretty easy to cover up research fraud.
How many scientists tailor their research and results so they can keep getting grants?
I don't know but one is too many, I think.
Criminalizing scientific fraud will lead only to criminalizing scientific dissent. Or should I call scientific dissent by its true name: heresy.
"And – and this was suggested by a commenter to an earlier blog post – not relying on scientific research for public policy purposes until it has been successfully replicated by someone else is not a bad idea." Sounds like something I'd say, and maybe I did! But the point is broader than that. The broader point is based on the observation that the vast majority of published science has no impact at all (not on public policy, not on textbook writers, not on reporters in the media, and not on other scientists). One approach to the replication crisis--which is popular--is to make sure that virtually all published findings (no matter how obscure and irrelevant) will replicate, even at great expense to the taxpayer. Another approach is to instead focus limited resources on published findings that gain traction, not just in terms of public policy but also in terms of media attention, attention from other scientists, attention from textbook writers, etc. NSF or NIH could fund proposals that make the case that the to-be-replicated results are gaining traction, so precious taxpayer dollars should be devoted to independently replicating those findings in a large-N study. That's a good reason to spend taxpayer dollars. Spending that money to instead ensure that everything replicates, not so much. Moreover, even reforms that seem free are more costly than they appear to be (e.g., in terms of a scientist's limited time). Just fund proposed replications of published findings that are gaining traction. It would solve so much of the problem. Unfortunately, NSF and NIH are both reluctant to do that. They'd rather fund "transformative" (often non-replicable) experiments. Fine, but put aside 5-10% of the federal science budget to replicate published results that matter. Why cheat if NSF is going to quickly fund an independent replication of the findings that are (momentarily) making you famous?
Despite the problems with fraud, I don't see how criminalizing fraud improves things.
Then why do we criminalize non-scientific fraud?
For all govt grants, require data to be archived in the money receiving institution And a copy in the US Library of Congress, And a copy with the org that publishes the study.
Set aside some 50% of grant money for replication studies, to decrease as the most cited studies are replicated or their non-replication is published.
Bounty rewards for those who find fraud in others’ work.
John Ioannidis has written extensively on this. One of his recommendations is to use P values of 0.01 instead of 0.05. That would definitely make P-hacking more challenging. Another reasonable response would be to not approve a drug or intervention until a 3d party meta-analysis shows benefit. We also need to figure out some way to remove big pharma funding of studies. Their incentives to cheat are just too great when they provide the funding for a study that if the results are positive will result in that company making billions.
Most journals require an analysis of possible study flaws in the discussion section, so that option is already in play.
Criminalizing science "fraud" would hamstring science, especially if the government was in charge of the analysis as well as the penalty. We've learned during the pandemic what Moderna and Pfizer will do to foist their vaccines on the public without appropriate study and analysis. Their ability to do that went straight to the bottom line. Many companies would not hesitate to invoke criminal intent on a rival company to protect market share and earnings.
Bioscience and medicine must more effectively monitor itself. Current practices at the NIH should be modified. Making one person or a small group responsible for the distribution of billions in grant funds to favored researchers is a practice that is an invitation to misuse. It most recently led to decisions that were illegal and resulted in the development of a virus that resulted in our most recent global pandemic. Clearly changes need to be made, and we in medicine must do it ourselves.
Glenn, you should continue to write on this, it's a great topic.