Why Science is Broken – Highlights

For those of us with an idealistic streak, we like to think of science as one of our purer institutions: Solely focused on seeking truth, where Spock-like people in white lab coats carefully sidle up to reality’s secrets, share them freely with the rest of us, and make our lives remarkably better in the process.

If only that were the whole story. Oh sure, scientists on PBS do their best to continue the idealistic narrative. But science today is a far different beast than it was even 70 years ago. A number of factors have conspired to take it off-course, and I have compiled the following partial list.

For most of the points below, a simple internet search will return plenty of results confirming the problem. In fact, for many of these, there is published meta-research that confirms it. Yet the pressures involved conspire to keep the problems alive, with no solutions in sight.

Money. Of course money is necessary for science to happen, but is also the chief problem. There are currently more scientists than the market can comfortably bear, so there is intense competition in science for funding. This leads to falsification of data (or just stretching the truth), and attempts to game the system in order to secure limited funding.

Money also influences what research can be done at all, because the source(s) will nearly always have a desired conclusion in mind before the research is begun. This is particularly pernicious in the pharmaceutical industry, where billions of dollars are at stake. A researcher who publishes results contrary to those desired by the funding source could easily find themselves out of a job.

Politically-correct conclusions are much more common than those that go against a prevailing cultural narrative. Research that agrees will have an easier time getting funding from the government or academia (the two guardians of what is politically correct), and will be more acceptable to reviewers when it comes time to publish.

Private companies frequently require researchers to delay or withhold publication in order to protect data that could form the basis of trade secrets or patents. This can keep negative findings away from other researchers, causing them to waste time and money that could be better used elsewhere.

Job security. Scientists are people too, with house payments to make and kids to put through school. Once a researcher has staked out a position in a particular paradigm, it is essential to stay in that paradigm in order to hold a job and get further research money. It would be career suicide to suddenly claim that one had been wrong all along. (This is a completely different dynamic than the scientific paradigms spoken of in Kuhn’s The Structure of Scientific Revolutions. Today’s paradigms are in more detailed areas of science, sub-fields actually, and live on based on funding. Kuhn spoke of major paradigms which really could be overturned based on counter-evidence.)

Consequently, “peers” reviewing papers for possible publication are ruthless in enforcing adherence to the current paradigm, because their jobs are also on the line if there is a paradigm shift. (Careers and reputations are built around being a major figure in a particular paradigm.)

Publish or perish. To get tenure and keep a job in academia, or to secure subsequent funding, it is necessary to publish, and quantity means more than quality. This pressure to publish is enough to cause researchers to alter or fabricate data.

Papers are far more likely to get published if they contain positive results than if they contain negative (i.e., that something doesn’t work, or that a previous paper’s results could not be confirmed). Positive results are far more interesting to readers, and less likely to rock anyone’s boat than negative results.

Peer review is broken. Reviewers sometimes squash papers that would get to press ahead of research they are working on. They will sometimes give a thumbs down to a paper by someone they don’t like if they can identify a competitor in their field. (Peer review is supposed to be anonymous, but authors can be unmasked somewhat easily.)  Two thirds of researchers have reported being asked by reviewers to cite additional works that had little or nothing to do with their paper.  This is a practice known as coercive citation, which allows reviewers to covertly boost ratings for their papers.

Some journals and conferences require submitters to pay to get a paper published, and then don’t actually submit the paper to honest peer review. Some conferences are shams, charging to let people speak, but having no checks and balances on what is presented.

Research papers are retracted far too often. (See http://retractionwatch.com/.) Whether papers needing retraction have always been out there, or whether more are needing retraction nowadays is an open question.

The replicability crisis. Researchers don’t seem to have enough controls on experiments to make them replicable in a reliable fashion. Problematic areas, such as medicine and psychology, have too many factors to allow accurate control of all of them. (Human beings are incredibly difficult to measure.)

Research that merely replicates other research is very valuable for establishing truths, but is far less likely to get published than research that plows new ground.

The p-value controversy. Even without fudging any of the data, an incorrect conclusion will be reached some small percentage of the time, just by luck. If you couple this with the fact that positive findings are easier to publish, you have a recipe for wrong findings to gain a foothold.

Sexual harassment, particularly of younger female researchers working under males. This is utterly reprehensible and inexcusable, but continues to happen.

Mid-career researchers, who have more to lose by not finding the results they are seeking, are motivated to overrule younger researchers working for them. Younger workers, more idealistic by nature would otherwise be free to reach less-biased conclusions. (Having said that, it is possible that researchers nearing retirement could be more trustworthy, as they have less to lose by pointing out major problems. Unfortunately, there is no good way to reliably employ this heuristic in judging research, as individual character varies too greatly.)

This list is probably not complete. Sadly, the points above mean that nearly any scientific conclusion published today deserves to be questioned. I’m not a luddite, and am not saying that we should ignore the conclusions science reaches. But we must be very careful when we latch onto any particular finding.