(Poll begins halfway down.)
I practice a type of psychotherapy called Transference Focused Psychotherapy (TFP) that a supervisor of mine once called “screw up therapy:” I screw up, and my patients tell me about it.
This doesn’t mean that I am actually screwing up – often I am, but screwing up is a complicated idea, and it’s rare for both parties not to have at least something to learn from bad communication. Rather it means that when my patients feel I have screwed up – by being a minute late, by forgetting their uncle’s name, by thinking they look sad when they feel anxious – I encourage them to air their feelings about my failings directly. This gives them a chance to understand and practice how they experience, work through, and resolve disappointment. I like this approach because in my view disappointment is the cardinal problem of clinical psychology, and learning to manage it, in all its subtlety, is the main goal of treatment.
Based on some of the comments I have received, my post about Jonah Lehrer’s WSJ article seems to have been my first screw-up moment with the internet, at least on any significant scale. My little blog used to get 35 visitors a day; that story got 3,000. But a lot of people didn’t like it, particularly my skimpy treatment of medians and geometric means. Chad Orzel’s post, in particular, summed up many of the critiques and made me realize that my main point had been lost somewhere.
For example, he thought that I thought the arithmetic mean disproved the whole WOC effect; I thought it just iced the cake of my main point, which is that in this particular study the geometric mean data wouldn’t have been very impressive to lay readers of the WSJ, and led Lehrer to choose a value from the median column. He felt I didn’t understand medians, while in fact I just can’t stand seeing them cited without skew data, because the statistical outsider doesn’t know that citing a median is, in science, often code for “this is a right-tailed distribution,” let alone even really know that there’s different philosophies about how to deal with the outliers that create such distributions. I skipped over that because it’s a topic of the future post I alluded to in the post – but he had no way of knowing that. He thought I didn’t understand that geometric means are similar to medians, as the methods section said, while my concern was simply that the methods section identified the the geometric mean as the outcome variable, and I’m old school conservative: I think reporters should report the outcome data identified by authors. About all we agreed on was that citing that 10,000 figure was an example of cherry-picking. But where he didn’t think that was my main point, I thought I’d made clear that it was the reason I’d twigged while reading the WSJ essay in the first place! I knew a cherry when I saw one, and believed further that it wouldn’t fly in a neuroscience lab, but would induce singsong demands by the presenter’s friends to “show us the re-eeeeeeeeest” – a teasing tune that implied exactly what I found: the variable in question – the median – was as a whole not as impressive as that one cherry-picked value. By the time I got to arithmetic means in my post I thought I was beating a dead horse; Orzel says he thought I thought I had come to my big reveal.
Most importantly, I worried he might think I was bashing Lehrer (though if anything I got complaints for revering him). I wasn’t. I’m pro-Lehrer per se. However the piece was really a sociological commentary – I was bashing a reporting system and a set of public expectations that doesn’t acknowledge that the neuroscience beat is exponentially harder to cover than the White House beat – to the point of it being impossible for any individual to know as much as needs to be known about the brain to explain it in a newspaper. What the public thinks they are getting from a “neuroscientist,” regardless of degree or writing ability, is much less than they believe; what is going on in that organ is far more wonderful and shocking than we yet understand. I was trying to point out the mistakes that can get made when too much pressure is put on one person – even as good as Lehrer – to explain a lot of different ideas at once early in his career, and on deadline.
Well, there’s one other thing I think we would agree on: my article seemed to prove my point. I was pitching it not at statisticians, but non-statisticians, and in the process I purposely skipped over complex ideas that make people’s heads spin – like log normal distributions – while slipping in my dislike of the slippery meaning of medians without owning it. Or maybe I just don’t understand medians. In any case, it was bad pop science – and thus was I hoist by my own petard.
Because Orzel was in good and thoughtful company I realized that at the very least I had a screw-up moment on my hands, and wanted some feedback. As in therapy, I figured it was time to stop acting-out reverberating disappointments, and have a check-in.
Below is a 2-minute multiple-choice poll. A quiz, really, since you have to have read Lehrer’s article. Each question asks about what turned out to be a disappointing point for at least somebody in my article. One of the answers is a summary of what I said in my post; the rest are answers suggested by critics, or merely plausible alternates. I’ve tried to identify every place critics think that I may have gone wrong, but if you feel I’ve gone easy on myself (a staple complaint in screw-up therapy, btw), please let me know and I will add another question. When the votes slow down (if they even come in fast int he first place) I’ll post the results, but you can also see them if you click on the link in each question. It’s not a WOC study!
If you haven’t already, to answer the questions make sure you’ve at least read Lehrer’s WSJ piece, possibly my postplus/minus comments, and for extra credit and a great single-serving critique, Chad Orzel’s discussion of my post.
Wikipedia defines cherry picking (fallacy) as follows: “Cherry picking, suppressing evidence, or the fallacy of incomplete evidence is the act of pointing to individual cases or data that seem to confirm a particular position, while ignoring a significant portion of related cases or data that may contradict that position. It is a kind of fallacy of selective attention, the most common example of which is the confirmation bias. Cherry picking may be committed unintentionally.”
Consider this quote by Chad by Chad Orzel, or his entire post if you have the time: ”…. this is the point on which the whole argument turns, Freed’s proud ignorance of the underlying statistics completely undermines everything else. His core argument is that the “wisdom of crowds” effect is bunk because the arithmetic mean of the guesses is a lousy estimate of the real value. Which is not surprising, given the nature of the distribution– that’s why the authors prefer the geometric mean. He blasts Lehrer for using a median value as his example, without noting that the median values are generally pretty close to the geometric means– all but one are within 20% of the geometric mean– making the median a not-too-bad (and much easier to explain) characterization of the distribution. He derides the median as the guess of a “single person,” which completely misrepresents the nature of that measure– the median is the central value of a group of numbers, and thus while the digits of the median value come from a single individual guess, the concept would be meaningless without all the other guesses.”