Science Is Self-Correcting ... Sort Of

August 16, 2011

One of the claimed advantages of scientific inquiry as a mode of acquiring knowledge is that it is a self-correcting enterprise.  For example, if someone claims to have discovered a process for cold fusion, that incorrect claim can be shown to be false or unwarranted by further research and scientific testing.  This rosy picture of the scientific enterprise suggests that scientific errors will be recognized and corrected, with false claims falling by the wayside during science’s inexorable march forward. 

The reality is a bit more complicated than that, principally because we humans are the ones carrying out scientific inquiry, and therefore, scientific investigations are at risk of being tainted by all the faults human flesh is heir to, including stupidity, negligence, and deceit. 

A recent article in the Wall Street Journal underscores both how prevalent errors are in scientific studies and how long it takes to uncover these errors.  Moreover, the number of retractions of scientific papers (retractions indicate an error has been detected) has been increasing rapidly this past decade.  Here are the numbers:  In the approximately 11,000 peer-reviewed scientific journals, the retraction rate has increased fifteen-fold from 2001.  There were only 22 retraction notices in 2001, but 339 last year and 210 through July 2011. 

The time that it takes for science to “self-correct,” that is, for the retraction of a flawed study to appear, is also increasing in length.  In 2011, retractions were published, on average, within about five months of the original study.  The average now is over thirty-one months. 

Three years may not seem like a long time to get it right, but if you have been taking medicine based on the original flawed study, that may be little consolation.  And, of course, in some cases it can take much longer.  The Journal article details the history of one flawed study that resulted in a significant change in the use of drugs to control hypertension.  Seven years later, after serious side-effects to some patients on the new regimen, the original study was shown to be flawed and possibly conducted unethically, without proper controls.

Perhaps most disturbingly, the Journal study reveals that about 26% of flawed studies are not the result of mere human error, but rather are the product of scientific fraud. In other words, a scientist has failed to follow proper procedures, for example, by cooking the data.  Why would a scientist engage in fraudulent conduct, especially if science is self-correcting?

Well, the motives could be many. First, the scientist may persuade herself that she is not doing anything wrong. She may be confident her theory is correct, so why does she need to follow all the necessary protocols? Her confidence may remain even if the data are not lining up quite the way they should. If she has the “right” answer, it’s not wrong to tweak the data is it? After all, once her theory is proven correct, no one is likely to notice or care (at least any time soon) that she has given the data a nudge or two.

Second, the scientist may hope that his improper methods are not discovered, or, if discovered, that they will be revealed only many years in the future. In the long run, we’re all dead, as many have observed.

And in the short run, one can obtain rewards, financial and otherwise, by making novel, important claims. Money and prestige are powerful motivations, for scientists as for anyone else. (Just make sure the results are not so extraordinary that they invite extraordinary scrutiny—as in the Korean human cloning scandal.)

Science is the best way to advance our understanding, no doubt. In part because of the success of science, ever bolder claims are made about its ability to provide us with important knowledge and guidance. Sam Harris, for example, has asserted that science can determine human values. Now is not the place to debate the validity of Harris’s thesis, but even if it were true, it’s one thing to know in the abstract what is morally right and wrong, and it’s quite another to adhere to those values in practice. Ultimately, science is only as dependable as the humans who apply it.

Comments:

#1 jerrys on Wednesday August 17, 2011 at 11:48am

There are several things wrong with this blog post. 

First there is the issue that the link is to an article that is behind a pay-wall and there is no indication of where to find the original report.  I presume the WSJ reporter was basing these numbers on some other research.  So my comments accept the numbers as reported here.  I have no way of evaluating whether there are problems in the way the numbers were obtained. 

Secondly.  Taking these numbers at face value there are at least two possible explanations of the increase in the number of retractions.  Possibility one is that the number of defective published papers has increased.  Possibility two is that editors have become more aggressive about forcing retractions.  The first is a problem, the second is an improvement. 

Finally, assuming each of the 11,000 journals published 100 articles a year a retraction rate of 339 is .03%.  This strikes me as a remarkably low rate.  I would guess that the rate of papers that reach incorrect results without doing anything wrong is much higher than this.  I think that far from representing a problem with “self correction” it reflects the health of scientific research.

#2 Simon (Guest) on Wednesday August 17, 2011 at 12:43pm

Oh and I think 2011 can’t be when things were better as it is urm 2011.

#3 John Dalby (Guest) on Thursday August 18, 2011 at 2:49pm

I’ll take science’s flawed, somewhat lenghty self correcting method over religion’s non-existant self correcting method.

#4 gray1 on Sunday August 21, 2011 at 9:29pm

One’s 15 minutes of fame will have expired long before the corrections come out anyway.  Advantage conferred, judgement deferred.  Harvard Ad Prac 101?

#5 Maz on Monday August 22, 2011 at 7:40pm

As time passes by more and more contradictions will occur, Science and Math are the answers but when we have the right result, and for that to happen, we must correct what we learn and there is always something new to learn and that can change all the neverending learning results.

Commenting is not available in this weblog entry.