Friday, September 15, 2006
and you thought astrosociology was dead
(graph from the Gerber and Malhotra paper. the expected shape of the graph in the absence of publication bias is like the right hand side of a bell curve)
Finally someone has written this paper. The graph is a histogram of the strength of results for a sample of articles on specific topics published in the two major journals of political science. See the dashed line? The dashed line corresponds to the p < .05 significance level that you need to be able to refer to your result as "statistically significant" and to be able to put a star by it. To the left of the line, no star; to the right of the line, star; this is the astrological base on which quantitative social science presently rests. The difference in height between the two bars is the difference between results that are just short of the significance-testing goal line and those that are just beyond it. The reason for the sharp difference in height is presumably the result of the tendency of journals toward favor positive results and the tendency for authors to favor analytic decisions that result-in-positive-results. The title of the paper is "Can Political Science Literatures Be Believed?", the drama of which I presume is intended to suggest to readers this is a very serious problem. In this assessment, I think the authors are correct.
Anyway, if anyone out there is (a) in sociology, (b) has time on their hands, and (c) thinks their career would benefit from a publication in a major sociological journal, one could assemble key coefficients from sociology studies (especially when the key coefficient is an interaction effect, which is a special class of perversities unto itself) and do exactly this analysis--perhaps along with its close cousin, known as a funnel plot--and probably score an AJS or ASR (and, at worst, something like Sociological Methodology). No original idea needed, although copping an attitude in the conclusion would probably be good. I can promise you that you will get results that lead to the same conclusion as this paper, and my suspicion is--especially regarding interaction effects--results will be stronger for sociology than for the political science literatures on which this paper focuses.
Postscript: The astrosociology site remains up and is here.