Someone's telling porkies

I was digging through a pile of old Natures, still in their mail-wrappers since 2014!  In the 4th December 2014 issue, there's an article and a News&Views on the cultural dishonesty of bankers. It had a certain resonance because I've just written a bit about codes-of-conduct whereby a group of people formally write down what they stand for and what is beyond acceptable limits. The primary research was carried out by a trio from the Dept Economics at the University of Zurich. It's the sort of study that has a good chance of making the cut at the weekly editorial meeting at Nature. As a general science publication, they have to include some copy that is accessible to a wide tranche of their readers - which is a very broad church from Astrophysics to Zoology. A neat little paper which plays on what everyone knows to be true - either to confirm our bias or to counter-intuitively rubbish it - has a chance.

It's a psychometrical study, such as economic researchers have been doing both before and after Freakonomics was published. They recruited 128 bankers from a multinational financial institution and split the sample into two groups: experimental and control. Each group was asked a series of questions and then put in a closed room and asked to toss a coin 10x and report how many heads came up. To encourage them to be bothered with such a trifling task the researchers incentivised everyone by saying they would be given $20 for every H that came up . . . provided that the total was more than the average score of a [fictional] pilot study. The punters were on a honour system to report the result accurately.  The experimental group's priming/prior questionnaire included questions about their work and work-practice in the banking industry while the control group were asked more neutral questions about TV and soccer.  Here are the results:

If you toss a coin ten times there is a very precise, symmetrical expected distribution based on the binomial theorem: 5H:5T is much more likely than 9H:1T.  This theoretical distribution is shown as blue bars in the histograms above.  The pink bars represent the distribution reported by N=67 controls on the Left and N=61 experimentals on the Right.  Do you think that the pink bars are shifted to the right=money-winning end of the distribution? Well, so do the boys from Zurich and the result is statistically [p=0.033] significant. (4 or 5)/61 of the experimental group report getting 10H:0T and winning $200 from U.Zurich's research fund. By chance alone you'd expect only 1/1024 to get a result so extreme. The neat thing about such studies is that you can't call any individual a liar but you can put money on the fact that someone(s) has been economical with the truth.  You might also wonder whether $200 was a sufficient incentive for a financial whizz kid who could pull that much down in 10 minutes by finagling the market in Thai Bahts.

The study was very widely picked up and reported by the world media: Les banquiers, tricheurs par culture (Le Monde); Banks breeding dishonesty (Sydney Morning Herald); Folk med dette job snyder mere end andre (avisen.dk); Bankers think they have to behave badly (Grauniad).  This reportage will doubtless have pleased the publicity department at U.Zurich and launched young Alain Cohn's career and Nature can be a bit of a whore for news-coverage, so they're happy.  And everyone who likes to put the boot in for bankers, which is pretty much everyone, is happy because their prejudice is confirmed.

But reflect. In science we use 1/20 or p<0.05 as a purely conventional level of statistical significance. If the chance of getting your result by chance is less than 1/20 then you may be on to something. But the corollary is that for every 20 such studies you carry out you can expect one result to be that extreme.  p=0.033 would certainly not be enough to convince your line manager that the company should invest $1 billion in developing a new cure for cancer based on your weeny pilot-study N=61+67 experimental result. If you were Brian Nosek you'd make the graduate student do the whole thing again in a different bank before rushing such a marginal result into print.  Or you could ask my friend Tony, who is a leader in his field because he has a good crap-detector to say v e r y  s l o w l y "extraordinary claims require extraordinary levels of proof". 

Explanatory note for those not brought up in Greater London (the entire readership?): Porkie is from cockney rhyming slang: pork pie = lie.  Similarly tit for tat = hat; so a hat comes a titfer.