If You Torture the Data Long Enough, It Will Confess

Ronald Coase? Irving John Good? Charles D. Hendrix? Robert W. Flower? Bulent Gultekin? Anonymous?

Dear Quote Investigator: Collecting and interpreting data is a delicate process that is subject to conscious and unconscious biases. The selective choice of inputs and statistical tests can yield results that are misleading. Here are two versions of a comical metaphorical adage:

  • If you torture the data long enough, it will confess.
  • If you torture the data enough, nature will always confess.

Strictly speaking these statements are ambiguous. Each interpretation hinges on whether the information in the coerced confession is correct or erroneous. The usual interpretations presume that the information extracted under duress is incorrect. Thus, torturing the data is counterproductive and not revelatory.

Both of these sayings have been attributed to Nobel Prize-winning economist Ronald Coase. Would you please explore this topic?

Quote Investigator: The earliest match located by QI appeared in an address delivered on April 22, 1971 by British mathematician I. J. Good (Irving John Good) at a meeting of the Institute of Mathematical Statistics. Good’s lecture was printed in “The American Statistician” in June 1972. Boldface added to excerpts by QI: 1

As Ronald Coase says “If you torture the data long enough, it will confess.” When data is tortured, it is useful when possible to reserve some of the sample for testing a hypothesis after it is formulated because there is not yet any satisfactory logic for using the whole of the sample.

Interestingly, Coase stated that he employed a different phrasing for the saying as shown in the citations presented further below dated August 1977 and 1982.

Here are additional selected citations in chronological order.

Continue reading If You Torture the Data Long Enough, It Will Confess

Notes:

  1. 1972 June, The American Statistician, Volume 26, Number 3, Statistics and Today’s Problems by I. J. Good, (Invited lecture at the 129th Meeting of the Institute of Mathematical Statistics on April 22, 1971), Start Page 11, Quote Page 14, Taylor & Francis, Abingdon, Oxfordshire, England. (JSTOR) link