Opinion When science gets it wrong

If only it were that simple. As a series of controversies over the past few months have demonstrated,science fixes its mistakes more slowly,more fitfully and with more difficulty than Sagan’s words would suggest. Science runs forward better than it does backward.

It takes a lot of time to look back over other scientists’ work and replicate their experiments. Scientists are scrambling to get grants and tenure. As a result,papers that attract harsh criticism may nonetheless escape the careful scrutiny required if they are to be refuted.

In May,for instance,the journal Science published eight critiques of a controversial paper that it had run in December. In the paper,a team of scientists described a species of bacteria that seemed to defy the known rules of biology by using arsenic instead of phosphorus to build its DNA. Chemists and microbiologists roundly condemned the paper; in the eight critiques,researchers attacked the study for using sloppy techniques and failing to rule out more plausible alternatives.

But none of those critics had actually tried to replicate the initial results. That would take months of research: getting the bacteria from the original team of scientists,rearing them,setting up the experiment,gathering results and interpreting them. Many scientists are leery of spending so much time on what they consider a foregone conclusion.

For now,the original paper has not been retracted; the results still stand.

Even when scientists rerun an experiment,and even when they find that the original result is flawed,they still may have trouble getting their paper published. The reason is surprisingly mundane: journal editors typically prefer to publish groundbreaking new research,not dutiful replications. In March,for instance,a psychologist at Cornell University shocked his colleagues by publishing a paper in The Journal of Personality and Social Psychology in which he presented the results of experiments showing,he claimed,that people’s minds could be influenced by events in the future,as if they were clairvoyant.

Three teams of scientists promptly tried to replicate his results. All three teams failed. All three teams wrote up their results and submitted them to the same journal. And all three teams were rejected — but not because their results were flawed. As the journal’s editor,Eliot Smith,explained,the journal has a longstanding policy of not publishing replication studies. “This policy is not new and is not unique to this journal,” he said.

As a result,the original study stands.

Even when follow-up studies manage to see the light of day,they still don’t necessarily bring matters to a close. Sometimes the original authors will declare the follow-up studies to be flawed and refuse to retract their paper. Such a standoff is now taking place over a controversial claim that chronic fatigue syndrome is caused by a virus. In October 2009,the virologist Judy Mikovits and colleagues reported in Science that people with chronic fatigue syndrome had high levels of a virus called XMRV. They suggested that XMRV might be the cause of the disorder. Several other teams have since tried — and failed — to find XMRV in people with chronic fatigue syndrome. As they’ve published their studies over the past year,scepticism has grown. The editors of Science asked the authors of the XMRV study to retract their paper. But the scientists refused; Mikovits declared that a retraction would be “premature”. The editors have since published an “editorial expression of concern.”

Once again,the result still stands.

But perhaps not for ever. Ian Lipkin,a virologist at Columbia University is doing what he calls “de-discovery”: intensely scrutinising controversial claims about diseases. Last September,he laid out several tips for effective de-discovery in the journal Microbiology and Molecular Biology Reviews. He recommended engaging other scientists — including those who published the original findings — as well as any relevant advocacy groups (like those for people suffering from the disease in question). Together,everyone must agree on a rigorous series of steps for the experiment. Each laboratory then carries out the same test,and then all the results are gathered together. At the request of the National Institutes of Health,Lipkin is running just such a project with Mikovits and other researchers to test the link between viruses and chronic fatigue,based on a large-scale study of 300 subjects.

This sort of study,however,is the exception rather than the rule. If the scientific community put more value on replication — by setting aside time,money and journal space — science would do a better job of living up to Carl Sagan’s words.