Association vs. Causation vs. Significance

Andrew Gelman has reviewed Distinguishing Association from Causation: A Background for Journalists over on his blog. He closes with:

My quick impression is that they’re promoting the best practices in statistical methodology, that all these companies are subscribing to. But there could be greater use of cheaper observational studies with better modeling (such as employing the propensity score approach, or even just better regression modeling) compared to expensive randomized experiments, and society might be better off as a result. Moreover, there is the issue of statistical versus practical significance.

He points us to Gall’s paper “Figuring out the Importance of Research Results: Statistical Significance versus Practical Significance” on the issue of statistical versus practical significance.

Clearly causality is a tricky issue in most studies, and I’ll grant that smart people are working on ways of sorting out causality and association (e.g., through the use of propensity score matching).

Yet, these days I’m also concerned about problems of practical significance – in the implications of inference based on models for decision-making. My concern derives mainly from teaching statistical decision theory to current and future public policymakers.

One consequence is that I’m surprised how little work has been done in the social sciences – especially in political science – on the role of loss functions in statistics. Why don’t we have a polisci/policy version of DeGroot’s Optimal Statistical Decisions – especially one we could use for training people in decision-making?