If Your Experiment Needs Statistics, You Ought To Have Done a Better Experiment

Ernest Rutherford? John M. Hammersley? Judy Campisi? Anonymous?
Dear Quote Investigator: The raw data collected in some scientific experiments is extensively processed via statistical operations. The tentative conclusions of this research may be accompanied with complex discussions of confidence levels.

The prominent physicist Ernest Rutherford preferred decisive experiments that did not require sophisticated statistical analysis. Here are three embodiments of this viewpoint:

  • If you need statistics, you did the wrong experiment.
  • If you need statistics to do science, then it’s not science.
  • If your experiment needs statistics, you ought to have done a better experiment.

Would you please explore this topic?

Quote Investigator: In October 1961 mathematician John M. Hammersley of Oxford University ascribed the third statement above to Ernest Rutherford. Hammersley was discussing Monte Carlo methods which are statistics-based methods used to construct efficient computer programs capable of generating approximate answers. Emphasis added to excerpts by QI: 1

In Monte Carlo work we can take heed of Lord Rutherford’s dictum: “If your experiment needs statistics, you ought to have done a better experiment.” In a sense, all good Monte Carlo work is self-liquidating: although we start out with random numbers in order to solve a problem, which may seem to be intractable by conventional numerical analysis, nevertheless we should strive to reduce their influence on the final result, and one should always seize any opportunity to replace a part or even the whole of the sampling experiment by exact analysis.

The evidentiary value of this 1961 citation is lessened by the fact that Rutherford died in 1937.

Below are additional selected citations in chronological order.

In 1973 “Biometrical Interpretation” by Neil E. Gilbert printed the following as an epigraph near the title page. The ellipsis is in the original text: 2

‘If your experiment needs statistics, you ought to have done a better experiment . . .’
Lord Rutherford

In 1977 “The Harvest of a Quiet Eye: A Selection of Scientific Quotations” compiled by Alan L. Mackay included the following entry. The supporting citation was dated 1967: 3

[Lord] Ernest Rutherford 1871-1937
If your experiment needs statistics, you ought to have done a better experiment.
in N T J Bailey The Mathematical Approach to Biology and Medicine 1967 (New York: Wiley)

In 1986 a syndicated cipher puzzle used the quotation as a solution: 4

PREVIOUS SOLUTION: “If your experiment needs statistics, you ought to have done a better experiment.” – Ernest Rutherford.

In 1992 biochemist Judy Campisi of Lawrence Berkeley Laboratory in California used an instance of the saying: 5

Campisi doesn’t believe in doing biology with statistical analysis of huge, anonymous populations of cells. “If you need statistics to do molecular biology, you need a better experiment,” she says. Instead, she and her coworkers in LBL’s Life Sciences Division study small populations of cells—no more than 100 or so at a time—and sometimes follow the fortunes of an individual cell from birth to death.

In 2006 “Statistical Analysis of Environmental Space-Time Processes” printed a variant expression: 6

If you need statistics, you did the wrong experiment. Ernest Rutherford

In 2012 “Smoothing, Filtering and Prediction: Estimating The Past, Present and Future” printed another variant expression: 7

“If your result needs a statistician then you should design a better experiment.” Baron Ernest Rutherford

In conclusion, Ernest Rutherford should tentatively be credited with the quotation in the 1961 citation. The ascription is weakened by the size of the interval after Rutherford’s death in 1937. On the other hand, the ascription is strengthened by the non-existence of substantive evidence pointing to an alternative creator.

Image Notes: Illustration of a statistical graph from geralt at Pixabay.

(Great thanks to Andrew Munro whose inquiry led QI to formulate this question and perform this exploration.)

Notes:

  1. 1962 August, U. S. Army Research Office (Durham), Report No. 62-2, Proceedings of the Seventh Conference on the design of Experiments in Army Research Development and Testing, Sponsored by the Army Mathematics Steering Committee conducted at U. S. Army Signal Research & Development Laboratory, Fort Monmouth, New Jersey, Date: October 18-20, 1961, Article: Monte Carlo Methods, Author: J. M. Hammersley (Oxford University and Princeton University), Start Page 17, Quote Page 18 and 19, Published by: U. S. Army Research Office (Durham), Box CM, Duke Station, Durham, North Carolina. (HathiTrust Full View)
  2. 1973, Biometrical Interpretation by Neil E. Gilbert (Institute of Animal Ecology, University of British Columbia), (Epigraph at beginning of book; unnumbered page), Clarendon Press: Oxford University Press, London (Verified with scans)
  3. 1977 (Reprinted 1981), The Harvest of a Quiet Eye: A Selection of Scientific Quotations, Selected by Alan L. Mackay, Section: Lord Ernest Rutherford, Quote Page 131, The Institute of Physics, Bristol and London; Crane, Russak & Company, Inc., New York. (Verified with scans)
  4. 1986 May 9, Arizona Daily Sun, Celebrity Cipher by Connie Wiener, Quote Page 21, Column 4, Flagstaff, Arizona. (Newspapers_com)
  5. 1992 Summer, LBL Research Review, Volume 17, Number 2, The Stages of Life, Start Page 32, Quote Page 32, Published by Public Information Department, Lawrence Berkeley Laboratory, Berkeley, California. (HathiTrust Full View)
  6. 2006, Statistical Analysis of Environmental Space-Time Processes by Nhu D. Le and James V. Zidek, Series: Springer Series in Statistics, Chapter 11: Environmental Network Design, Quote Page 185, (Epigraph to chapter 11), Springer, New York. (Google Books review)
  7. 2012 Copyright, Smoothing, Filtering and Prediction: Estimating The Past, Present and Future by Garry A. Einicke, Chapter 2: Discrete-Time Minimum-Mean-Square-Error Filtering, Quote Page 45, (Bottom of the page), Published by (InTechOpen) InTech, Rijeka, Croatia. (Google Books Preview)