Charlie Munger, chief investment advisor to Warren Buffett, famously said, “Show me the incentives and I will show you the outcome.”  In the academic world, it has always been about publish or perish for researchers.  Although there is an academic plague of marginal scholarship, publications are necessary for promotion and for the sake of professional reputation.

Our recent work with the National Association of Scholars critically looked at three surefire publication topics — all introduced at some point by Harvard University researchers.  This offered us excellent insight into what the “incentives” of their research really are.

The first topic is the food frequency questionnaire or FFQ.  You ask a cohort of people what they eat and how much of each food.  The initial FFQ was developed by Harvard researchers in 1985 and asked about 61 foods.  Next, you wait for some time and then ask the people about their health conditions.  You collect other variables that might influence these conditions, such as age, sex, body weight, etc.

Analysis of the data collected is a mix-and-match of food and its quantity, health conditions, and demographics.  There are thousands of ways to analyze the data, so literally any answer you want is possible.  There are methods to adjust the analysis to address the multiple testing problem, but the Harvard researchers (and others) do not adjust their analysis.

Other academics quickly realized that FFQ research and getting any answer you want was a surefire path to publication.  Now, 40 years later, there are over 50,000 FFQ studies in published literature, and arguably none of them is reliable.  They are all untrue!  Original Harvard research doesn’t sound like much of an incentive to us.

The second topic, a study of six small U.S. cities by Harvard researchers in 1993, was about air quality and health effects.  Several historical instances of severe air pollution caused by cold, temperature inversions, small particles, and acid in the air leading to human deaths cried out for explanation.

The Harvard six cities study, funded by the government (EPA), pointed a “statistical correlation” finger at small particles and their major constituent, sulfate.  A second (larger) study, also funded by the EPA, was negative.  The EPA dropped the researchers of the second study and continued funding Harvard and like researchers.

There are now millions of studies on air pollution in the literature.  However, it is well known that the six cities and like studies in literature are plagued with methodological problems.  What kind? you ask.  Use of questionable research practices, multiple testing bias, irreproducible (false) research claims.  Recognize a pattern?

A third surefire topic for publication was the claim that the Implicit Association Test (IAT) — a method perfected by a Harvard graduate — captured implicit bias, subconscious bias.  Implicit bias was claimed to explain race-based, sex-based, etc. behavioral differences in society.  Everyone is biased according to the IAT.

There are over 20,000 articles in the literature on the IAT.  However, careful analysis by psychometricians and others has found no correlation between IAT scores of individuals and their actual behavior.  A simple explanation is that a name was initially assigned to a measurement without validation.  Additionally, statistical analysis shows that the IAT adds little to predictive power over other information.

The three topics share several characteristics: government-funded research, an easy path to publication, researchers painted as virtuous, scores of publications in literature over time, and of course there is considerable evidence that the claims are not valid.

As for the incentives of original Harvard research, we’ve seen the past, and it is horrible.  How could the future be at all different unless something changes?

S. Stanley Young, Ph.D. is the CEO of CGStat in Raleigh, North Carolina and is director of the National Association of Scholars’ Shifting Sands Project.  Warren Kindzierski, Ph.D. is a retired college professor (public health) in St Albert, Alberta.

Image via Pixabay.