Given the current emphasis on “believing in science” (a meaningless category of belief), I can’t recommend highly enough Zoltan Dienes’ exploration of scientific inference – Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference.
He gives a decent enough survey of the problem of first of all deciding what is science, although within the framework of how we might decide that experimental results constitute evidence of a theory that we can use to infer.
He begins with Karl Popper’s philosophy of science and, in particular, the notion of falsification. He then takes us via Kuhnian paradigms and the notion of theory approximation. This latter subject ought to be of great interest in an age where it seems we are awash with “theories” of all kinds, not just scientific ones. For example, there are myriad “theories” of how enterprises function, although it is doubtful that many of them constitute a theory at all. As Karl Weick pointed out – the products of laziness and great effort can often look the same: references, data, lists, diagrams and hypotheses.
The book then explores the inference approach of the classical Neyman/Pearson statistical framing (i.e. the t-test et al) but in a way that explains the validity of the approach, or not. This is the framing that nearly all papers use when purporting to possess statistically significant results. I am guessing that even many well-educated folks have never paused to ask what, exactly, is significance? It is clear from at least my daily dose of Twitter that our collective understanding of statistical inference is nearly zero.
He then goes on to explain Bayesian inference, a topic that is seemingly straightforward, yet deceptively complex in its subtleties. The ability to assign a probability to a future event based upon observed data is one of those things that ought not to work, except that it does — there would be no cellular communications without Bayes. On the other hand, does it really work? I am not sure I know the answer.
However, if there is a branch of mathematics that might allow us to say that we “believe” in something, then Bayesian statistics makes strong claims about beliefs.
The book ends with an exploration of likelihood, a measure that many confuse with probability. As an aside, I would say that the mechanism known as a Maximum Likelihood Estimator (MLE) is something that any well-educated person ought to know. Indeed, I will add it to my list of important things to know, if I can find it. (It is a list that I made when thinking about my children’s education.)