Jonah Lehrer is Not a Neuroscientist

May 29, 2011

Psychotherapy

Screen shot 2011-06-07 at 8.53.47 PM

================================================================

* Note – the man himself, Jonah Lehrer, comments in the comments section!

** Brutal Chad Orzel critique here

*** Poll about whether I am indeed an idiot here.  

**** Kinder, gentler Nicholas Carr critique here.

***** Beginning of 3-part post emphasizing that statistics are not scientific – the mysterious second stage of the critique alluded to below –  begins here.  Light editing and side notes to reader re: my statistical interlocutors added 6/7/11.

****** Galton’s original article here

Thanks for the hullaballoo!

================================================================

Jonah Lehrer is not a neuroscientist.   And I am not the first to admit it.

Responding to a mention of Lehrer in a June 7, 2010 Discovery Magazine review of Nicholas Carr’s The Shallows: What the Internet Is Doing to Our Brains, a reader said the same exact thing.  It was – apparently – news to the story’s author:

Moseman, the author whose fix involved changing the wording about Lehrer (the article now identifies him as a “neuroscience author and blogger”), can be forgiven for his mistake. Our society badly wants help understanding the relationship of the mind to the brain, and we appear to have nominated Lehrer to be our man.  He is young, he is handsome, and he is a certified smartypants.  Fresh off a Rhodes scholarship he started writing about the brain and hasn’t stopped.  He can’t be more than thirty, and already has two bestsellers on the topic: Proust Was a Neuroscientist and How We Decide.  I am sure another is on its way.

And yet, as Katharine X pointed out back in 2010, he is not a neuroscientist.

Now I must confess something.  Though I don’t pay much attention to the popular writing on neuroscience, I know Lehrer is one of the stars of the pop neuroscience world.  Probably most people reading this do as well.  And so I know that the title of this post – which is, in truth, merely the search phrase I typed curiously into Google last night after finding problems with an article of his – may seem provocative.

Merely stating that Lehrer is not a neuroscientist, which in my defense is as empirical a fact as ever there was (Lehrer has a bachelor’s degree in neuroscience from Columbia, but did not go further in his studies) may be seen as an attack: some sort of dig, or put-down, or attempt to call him out.

Given that Lehrer has never claimed to be anything but what he is, which is a journalist and public intellectual, it can’t be that Lehrer himself will be insulted. He, of all people, knows he is not a neuroscientist!

No, if I can pull a shrinky move here, it’s my guess that it will be his fans who will be upset.  In saying that Jonah Lehrer is not a neuroscientist, I am messing with their fantasy.  You see, the general public has wanted Lehrer to be a neuroscientist. And by this they haven’t meant they wanted him to have a PhD.  I honestly don’t think they care about his degree.  They want him – and anyone else they give the honorific – to understand the brain and then explain it. Pleasantly.

They want neuroscience to make as much sense as Jonah Lehrer’s writing does, and the brain to be as unthreatening as Jonah Lehrer makes it.  He gives them hope that someday they’ll understand why they are bathing their neurons in all this alcohol and nicotine and caffeine and chocolate and Xanax and Prozac, and how they could better fiddle with whatever knobs are in that thing to bring on more and better happiness.  And they want it to be like buying an iPhone.  They want it to be intuitive and fun.

You might say that that utterly brilliant title – Proust Was a Neuroscientist – captured the public’s hope that, in the end, understanding the brain would be as easy as lying in bed and remembering that madeleine. Lehrer’s title said it, and we followed him down the rabbit hole: if Proust could be a neuroscientist then, by simple logic, the brain must be a piece of cake. 

Phew!

And yet.

It surely has not escaped the notice of those of us who care deeply about the brain that in the final analysis Proust – that great introspectionist – may not, in fact, have actually been a neuroscientist.  Sitting and thinking may not be the same as gazing at neurons under a microscope.  And Proust’s folk psychology, which we all share – our concepts of memory and emotion and cognition and volition and action and so forth – may not in fact be up to the job of figuring out what makes us tick. And it has not further escaped us that if Proust was not a neuroscientist, then in turn perhaps Jonah Lehrer, for all his gorgeous writing and enthusiasm, is not a neuroscientist either, and that deprived in this way of all our guides we have perhaps been left all alone with this strange, even alien, organ in our heads.

This is, I think, the question that – if we are honest – haunts us all.

And so it was last night, as I lay on the living room couch in my mother’s house, after she and my wife and son had gone to sleep – the couch on which my father used to lie, and where he would read and think and snooze – with such thoughts hovering vaguely in my head as some potential, waiting to coalesce around some grain of sand in the world, that I came upon an essay published yesterday in the Wall Street Journal, in which Jonah Lehrer discussed with some bravado the implications of a recent scientific study of the wisdom of crowds for the American way of life.

Lehrer had a lot of interesting things to say – the main point of which was that a recent study seemed to show that a crowd of people makes better decisions if its members don’t talk to each other than if they do.  But I had a problem – an unexpected problem.  I didn’t believe it.  I didn’t know what I didn’t believe, but whatever was going on in that article, it was unbelievable.  My proverbial B.S. meter was going off like gangbusters, and I had no idea why.

Now as Malcom Gladwell discussed so memorably in the opening pages of Blink, I cannot reconstruct why I sensed something wrong. Nevertheless, to purposely mix metaphors, I felt annoyed by an intellectual tickle I could not scratch.  So I decided to do a little detective work, lying there on my dad’s couch holding an Apple, before retiring to bed.

Looking back, the investigative impulse may have come from the fact that Lehrer never identified the article he was discussing, referring to it simply as “a new study by Swiss scientists.” I was a bit peeved that the Wall Street Journal did not feel compelled to credit the author of an original article, or give the title of the original article, or even state the scientific journal in which the article was published. Perhaps the scientist in me – who well knows how long it takes to produce these damn papers – wanted to give that faceless person in Switzerland his or her due.

Or perhaps it was that Lehrer’s opening paragraph had left me in hysterics: “America depends upon the wisdom of crowds. When voting, we rely on the masses to pick the best politicians. When investing in stocks, we assume that, over time, people will gravitate toward the best companies. Even our culture is increasingly driven by the collective: Just look at ‘American Idol.’ “

I had almost fallen off the couch. I trusted that Lehrer knew his examples were all deliciously insulting, coming as they were one after another like rapid-fire one-liners in a stand-up routine.  I loved the in-your-face subversion and reckless disregard he gave his safety in squirreling these digs into the conservative Journal in what surely was a sort of intellectual hi-hinks, or prank, or punk’ting, or whatever it is we are meant to call practical jokes these days.

Bottom line: he was pwning the Journal.  For of course, our nation’s recent selection of Bush, twice – once? – and our support for the Iraq caper and the housing bubble and, worst of all, Lee DeWyze, have all pointed to a single conclusion: As a nation we may be many things, but wise surely is not one of them. We are an empire in decline, despite the repeated warnings of those who would save us.  Lehrer knows this better than anyone, surely, having studied abroad.  I can only imagine the crap he took on our behalf from those Oxford wits

For anyone who takes umbrage at this portrait of our national discretion, I have a visual retort:

Listen, I am as mesmerized by Snooki as the next American. I once saw her interview J-Wow on the red carpet on live TV – I think it was a Dairy Queen opening – squeeze J-Wow’s implants and then say, thoughtfully, “looking good tonight.” I could not turn the channel.

I do not chalk this up to wisdom.

But if I had to say what the clincher was – what sent me off looking for what was making me blink, it had to be two stray words: for instance.

Here’s the quote where I found them.  Lehrer is early into his article and describing the study upon whose data, at the piece’s end, he would base his sociological observations and policy recommendations.

The experiment was straightforward. The researchers gathered 144 Swiss college students, sat them in isolated cubicles, and then asked them to answer various questions, such as the number of new immigrants living in Zurich. In many instances, the crowd proved correct. When asked about those immigrants, for instance, the median guess of the students was 10,000. The answer was 10,067.

And there it was.  Two words that triggered my blink moment.  For instance.

Now to understand why these words were such a trigger, you must know that for the past seven years I’ve wrestled with fMRI-generated matrices and endless columns of subjective psychology data, all of which I’ve analyzed up the wazoo.  I have waded through waaaaaaaaaaaaaaaaay too much data – or shitty fucking data, as we call it in the biz – to take Lehrer’s sentence at face value.  Which is to say, I would never believe that a guess – by 144 people, no less – less than 1% away from a correct answer could possibly be a “for instance.”

To say this sentence – “for instance, the median guess of the students was 10,000. The answer was 10,067″ – was, like hearing him say “New York City public transportation mixes all classes into a single melting pot. For instance, Kim Kardashian sat next to me on the subway yesterday.”

That’s wouldn’t be a “for instance.” That would be a “what the f***.”

But that wasn’t the only funny thing about that 10,000. I’ve run way too many SPSS statistical analyses to think that an average guess by 144 people could be a round number – eg, would end in 0.  Let alone 0000!  And so part of my blink was the thought “10,000 – jeez, what kind of miraculous average is that?”   

Now if you have already seen my error – for Lehrer’s conceptual mistake is buried in his sentence, for all the world to see – bear with me.  It was late at night, and I had been pulled in by the way in which he had framed his story, and so I was slow to pick up one crucial word.  And if you haven’t seen my error yet, join the club – and wait a moment more.

With all of this bad statistical juju running through my head, the part of me that knew Jonah Lehrer is not a neuroscientist – and therefore might have made a mistake in reading this article and then waxed philosophical about his error – blinked.  I blinked, and then I opened up a new tab in my browser and went searching for the original article to see what on earth was wrong.

The problem was, the original article wasn’t named in the WSJ piece – a breach of scientific etiquette I hope the editorial staff fixes in the future. Luckily a commenter on Lehrer’s piece, also noting the Journal’s oversight, had done some detective work, and provided the link.  Here it is:  Lorenz et al (2011) How social influence can undermine the wisdom of crowd effect. PNAS. 

I opened the paper and did what I always do – skip the intro, go straight to the tables and figures, and then to the methods.  If you ever read a science paper, you should do the same thing yourself. Reading intros and conclusions first is for suckers – they can say anything the author wants, and reading them allows the author’s “spin” – as we scientists call intros and conclusions – to frame your analysis of the data.  In science, the only thing that matters is the methods and the data, because it’s where the author can’t hide behind a spun story.

And on page three, there it was.  The problem.  In around sixty seconds, no more than ninety, I found Lehrer’s 10,000 and then his 10,067 and immediately saw what he was doing and why it was making me blink.  It was in the table just below – Table 1 – the only table I’ll talk about in this post.  The red circle around the number 10,000 – the 10,000 that Lehrer used as his for instance – is drawn by me, because it’s the heart of Lehrer’s and the Journal’s problem.  Or rather I should say cherry.  And it is a huge, huge problem cherry.

Now listen.  In the next few paragraphs I am going to start with this 10,000 and use it to explain why Jonah Lehrer should never, in a million years, have used this number as his “for instance” – and further why the table as a whole implies the very opposite of the point he made in his article.  Which in turn renders his patriotic conclusion invalid. In a future post – coming in a few days – I will begin to discuss deeper conceptual problems with Lehrer’s article involving something fancy sounding – the metaphysics of central tendencies.  But in this post I am going to stick to examining just this one error, because of its importance to our thinking about the whole field, and the whole trade, of pop neuroscience.

I’ll try to make it fun.  Think of the next few paragraphs as an episode of CSI, except that the crime is the misreading of a single Table in a science paper, and David Caruso is – sorry Caruso – me.  Oh, and nothing dies.

First, take a good long look at Table 1.  Seriously. I’ll reprint it to make it easier.

Now listen: most non-scientists see a table like this and freak out.  They take around 3 seconds to decide they can’t understand it, get scared of feeling stupid in the face of all those numbers, and so they calm down by skipping over it and back to the words.  Scientists have a huge advantage over their non-scientist friends on this front: they don’t expect to understand this table in three seconds. Or even three minutes.  They look at it the way a piano player might look at a Bach score, or an art lover might look at the Mona Lisa.  They look at it for a good long time, lingering with their eyes over the columns of numbers, and getting a visceral feel for it.  The table becomes a living thing for them, with a personality.  And only after they have a little bit of a vibe from the table do they start trying to understand all the column- and row-headings. Do the same.  Allow the numbers to form some vague impressions in your mind.  Do they have decimal endings?  Are they all even or odd?  Are they short or long?  Is there lots of variation between them?

This vibe in hand, let’s begin.

Look at the red circle.  That 10,000, there in the last column, third row – it’s just crying out to be compared to the “true value” in the second column, third row – the 10,067.  Let those two numbers resonate with one another, as they surely did in Lehrer’s mind.  See how similar they are?  To quantify this, the authors calculated the percent similarity (the guess minus the true answer divided by the true answer – or 67 divided by 10,000.  The percent difference between them is 0.7%, as indicated by the 10,000(-.7%).

These numbers – the ones Lehrer cited in his article – will be your anchor points in the table.  Always go back to them if you get confused.  In an English sentence, a scientist would say that this line reads: “the 144 Swiss college students’ median guess as to the number of new immigrants to Zurich was 10,000, a difference of only 0.7% from the true value of 10,067.”

But instead of considering this value an “instance”, as Lehrer did, you should notice that this was by far the best guess produced by “the crowd” in response to the six questions they were asked. The fact that Lehrer chose the third value in the column, rather than the first, is also a bit of a giveaway that there was a problem with the first two – at least if you wanted to make a wise-crowd point.

Running down the right-hand column labeled “median,” the other guesses by that crowd of 144 college students miss their targets by a whopping 29.3%, 59.1%, 0.7%, 14.1%, 60.9% and 56.9% respectively, for an average error of 38.5% away from the correct answer.  That’s a big error. That ain’t no 0.7%.

So here’s the first question that we want to ask the science editor of the Wall Street Journal: why did they let Lehrer report only one value – the 0.7% – and say “for instance” instead of reporting the group average – the 38.5%? But we don’t have to ask.  The answer is obvious.  What he did is an example of a fallacy in science called cherry picking.  Which is to say, that 0.7% made the point about the wisdom of crowds in spades.  It was almost as good as Galton’s initial, amazing Wisdom of Crowds finding.  Which is to say, it made a good story.  And if I didn’t understand statistics, I would have acted like a kid in a candy shop and cherry-picked that number too.

We’re getting close to my error now.  Because as you get into the groove of analyzing this table, something else weird and blinky should jump out at you – the same thing that jumped out at me when I saw the 10,000.

All of the median numbers end in 0, and none have decimal points.

Wait. Isn’t this article on the wisdom of crowds?  The median “guesses” to the six questions were 130, 300, 10,000, 170, 250,  and 4,000 respectively.  The correct answers, meanwhile, were 184, 734, 10,067, 198, 639, and 9,272.

Okay. Stop a second.  If you had to decide which of those two sets of six numbers represented a group average guess – or as Lehrer put it in the article, “in many instances, the crowd proved correct. When asked about those immigrants, for instance, the median guess of the students was 10,000. The answer was 10,067″ – which set would you say a “crowd” would come up with?

Obviously your answer would not be the numbers ending in the 0’s.  What are the odds that 144 people are going to guess 144 different numbers and the average of those numbers will end in 0? One in ten – and only if the original number isn’t odd, in which case there should be a decimal. But of course the numbers are “funnier” than that – there’s that 10,000 and that 4,000 and that 300 and that typical number 250. These numbers shouldn’t be popping up for a 144 person average.

It was as I mulled this over that my eye went up to the word “median” at the top of the column, and I realized with a “duh” what was happening. And if you have been bothered, as you read along, by my apparent confusion of the difference between mean and median, this was the moment where I realized my mistake.  I had overlooked the word median, which seemed to be used incorrectly, and gone with the gist of Lehrer’s implication that he was talking about a group average – as when he said “the crowd proved correct…” – as an article on the wisdom of crowds would lead one to assume, and as the article in question overtly states is the most common measure of crowd wisdom.  It had never occurred to me – as it never occurred to he study’s authors, who identified the geometric mean as the outcome variable of interest – that Lehrer would be talking about an actual group median. But now that I realized he really meant median, and that maybe he didn’t know what median meant.  Because median guesses are not guesses by a crowd, as Lehrer states.  They are guesses by a single person. They are guesses by the median person in the group – number 72 (or  72 and 73 in even samples) out of 144.  Which is to say, they have nothing to do with the point of the Journal’s article!  [Note: this is the section that drove all my critics crazy.  I had implied, by not overtly said, that I was saving my critique of central tendency junkies for a group of later posts; they had little way of knowing this.  My bad.]

In statistics, the medoid is defined as the person with the middle value in a group – half the group is above them, and half below; medians are essentially equivalent to the medoid.  Which means that, for all intents and purposes, the median person is one guy.  Not a crowd! A single person! Which is why the numbers are so pretty – they were chosen by individuals. [Again, people went nuts over this - see this series of posts for the metaphysics behind this point; it is NOT (and cannot be) empirically wrong, as statistics - like mathematics - are not a branch empiricism. They are a branch of applied metaphysics.]

Feeling silly, I looked at the table more closely, and saw that the words “wisdom of crowd aggregation” at the top of the table explicitly excluded the median category. This was a bad sign for median – it meant the authors probably just included it for kicks, and not as a measure of crowd wisdom – which made sense, given it wasn’t a crowd answer.

And then I read the methods section, which confirmed my reading in spades. There I found this sentence: “this confirms that the geometric mean…. is an accurate measure of the wisdom of crowds for our data.” [Lorenz et al, p.3, top left paragraph].  Only later, in the discussion, do they mention that medians are close in value to the geometric mean, but fail to explain why they did not use it. One has one’s suspicions……

Things were looking bad – really bad – for Lehrer’s characterization of the paper. Not only had he cherry picked a single value out of a column of values, but he had chosen a column of values the authors did not use as their dependent measure; the authors explicitly say that it is the geometric mean that should be used to judge crowd wisdom.  Not the median!  In his WSJ article Lehrer is using a number generated by one person, to one question, instead of by 144 people to 6 questions.

[Again, critics went nuts over this, saying that the median is a group statistic.  First, this is an article of faith - statistics are not an empirical science - and second, most readers of the WSJ would not understand this meta concept, and would recognize median values for what they are - analogs of a representative democracy, in which each value along the x-axis turns out to have one vote, regardless of what that vote is for, and only the the one or two middle members of the group go on to represent it - with all information about outliers thereby, purposely, lost. This somewhat complex critique is explored in a later group of posts.]

Furthermore, the authors explicitly tell him that he should be looking at another column of numbers!  As my grandfather used to say, Oy!  It’s as though Lehrer either didn’t read, or didn’t understand, the table and the methods section. Unless, of course, he purposely mischaracterized it – which I sincerely doubt.

Now of course Lehrer’s mistake was a relief for me, because it explained why I was up surfing the internet looking for Swiss college students doing strange things in cubicles when I could be sleeping.  It meant my blinky hunch that something was wrong with Lehrer’s article was correct.

The bad news was I was hooked.  Now I wanted to actually understand the paper that Lehrer had misread, to see if despite his error it nevertheless said what he claimed it did.  I didn’t know yet whether Lehrer had made a small technical mistake or a big conceptual one.

In doing this, I should make another comment about the psychology of reading science papers.  If I were the kind of person who trusted what authors say about their own data, I would never have been in this position in the first place.  The introduction to the paper, which I did eventually read, more or less supports Lehrer’s interpretation.  But as a scientist – having been on the inside of the “spinning wars” that roil the field – I was as cynical as anyone. I was completely uninterested in what the authors said their data said.  I kept looking because I wanted to know if it really said what they said it said.

So I kept looking at the data Lehrer should have been looking at. In the figure below, I’ve circled that data – the exact data the methods section said we should be looking at.  The geometric mean (circled in blue, below) – whatever in God’s name that is. [Chad Orzel got upset over this throwaway line's effort to spare my readers a conversation about the philosophy of tail reduction that, as implied, I was saving for a longer discussion in later posts. Sorry Chad!]

When I looked at that middle column of numbers, labeled “geometric mean,” I saw that those numbers look horrible.  Of the six guesses, the closest to the true value – row 4 – is 11.9% off, not that gorgeous looking 0.7%, and three of them – rows 2,5,6 – are over 50% off.  And then I started thinking doing what I thought that I might do – I started doubting the study’s own authors’ portrayal of their results.   This crowd of students, I thought to myself, is not looking too wise after all.

And then I noticed that very first column of numbers (green circle, below). The one under the words “arithmetric mean,” which is just a fancy-pants way of saying “the number that the students actually guessed.”  This is the number that Galton originally reported, according to Wikipedia, when introducing the world to the Wisdom of the Crowd phenomenon.  That is, this is the number that real people literally wrote down when guessing the answer to the question.

And if you thought the geometric mean was bad, the arithmetic mean - unlike in Galton’s study – is horrrrrrrrrrrrrendous.  Here the crowd was hundreds of percents – yes, hundreds of percents – off the mark.  They were less than 100% off in response to only one out of the six questions! At their worst – to take a single value, as Lehrer wrongly did with the 0.7% – the 144 Swiss students, as a true crowd (unlike the 0.7%), guessed that there had been 135,051 assaults in 2006 in Switzerland – in fact there had been 9,272 – an error of 1,356%.

Hell, Snooki could have done better than that.

I won’t trouble you with any more of the methods section, save to say the authors tie themselves in knots explaining why the actual guesses people make – the numbers they write down and actually mean – are not a good estimate of the wisdom of crowds, while the geometric mean – something confusing involving outlier-killing logarithms – is much better.  Suffice it to say that none of the actual students writing down actual answers would have recognized their distorted numbers after such manipulation, so that whatever wisdom such a crowd might have, none of its members would know it.

Just put that aside.  We’re going back to Lehrer’s article. Consider this early quote:

Here’s the bad news: The wisdom of crowds turns out to be an incredibly fragile phenomenon. It doesn’t take much for the smart group to become a dumb herd. Worse, a new study by Swiss scientists suggests that the interconnectedness of modern life might be making it even harder to benefit from our collective intelligence.

Wait. What?

Having just analyzed the original article, you should be left wondering what Lehrer means when he talks about “the smart group.”  Forget the dumb herd part – I don’t even know if there are problems there; the article came out yesterday and I am on vacation and squeezing this post in between family outings.  I’ll try to revise later.

But the problems with Lehrer’s premise are enough for me.  The authors of this article collected raw data that make it quite clear that in this particular paper there is no smart group.

There is no wise crowd!  Those Swiss students blew it.  Blew it! Every single question, the arithmetic mean, and the geometric mean, and the median – save for a single case that Lehrer cherry-picked – was from a human standpoint wrong, wrong, wrong, wrong, wrong and wrong. The end.

So what is Lehrer talking about? He is talking about that single 0.7% single-person data point: one person, selected after giving their answer, got close to the correct answer on one of six questions.  One person guessed 10,000 when the answer was 10,067.  That’s one hit out of 144 x 6 = 864 attempts.  That seems about right to me, from a common sense perspective. Which is to say, that is a shitty batting average. And so the actual numbers produced by actual people give the lie to all three of his examples at the start of the piece, about elections and stocks and American Idol.  They explain, if anything, why stock markets get things wrong, America elects bad presidents, and the winner of singing competitions is never the best singer – with or without social media and its supposed herds.

Which brings us back to the central concern of this piece: what does it mean to be a neuroscientist?

Here’s my deep point. I don’t care about straight psychology – straight psychology is, not to pull punches, over.  I care about neuroscience. And Lehrer was not trying to be a neuroscientist in this article.  This was a straight-up psychology article.  But modern neuroscience, his chosen wheelhouse – particularly the subfields of behavioral and affective and cognitive and social neuroscience – is radically more complex than straight psychology.  Its experiments are like this study combined with brain imaging.  Similar and more serious mistakes, by scientists and their interpreters, might be made concerning the psychological interpretation of the activity in various parts of the brain.

In short, neuroscience is really, really complicated.  So why are we letting our fantasy that understanding the brain might be a piece of cake lead us to expecting any one person – even a Rhodes Scholar – to figure it out?

One person can’t do it.

I think we’re at the end of anyone seriously thinking they, alone, can understand the brain.  None of us is a neuroscientist. Not Jonah Lehrer, not me, not even Antonio Damasio.  It’s going to have to be a team effort from here on out.  Nobody will ever understand the whole brain – conceptually, maybe, but not at the neuroscience level – the level of its physical structure.  That sucker has 100,000,000,000 neurons, each connected to 10,000 others, each firing around 100 times a second.  That thing is exponentially harder to understand than any other phenomenon in science – some people say the universe.

And that’s where we, the consumers of pop neuroscience, need to get real. Really, really, really real.  I admitted Jonah Lehrer is not a neuroscientist at the start of this piece, and I’m sure he admits it, but now we all have to admit it.  We need to make him – with all his ambition and intelligence and thoughtfulness – not be a neuroscientist on our behalf.  No single person can be a neuroscientist ever again – not the way the public wants them to – not the way a dermatologist can still be a dermatologist, or a carpenter a carpenter.  Too many smart kids, in too many labs, with too many bright ideas, and too much government and industry financing are discovering too many new ideas for any one of them to keep track of all the facts of the brain.  There is simply waaaaaaay too much to know.  We’re going to have to do this together.

One of the hardest parts of scientific training is the surrendering of hope – in particular, the hope that one person can understand it all.  All scientists go through this, more or less.  There is simply too much information, too much technical knowledge, too many specialized concerns, for anybody to understand all of neuroscience, or psychology.  I cannot review papers in cellular neuroscience, for example, and will not review technically demanding papers in neuroimaging. And I know jack – officially, at least – about the academic field I care most about, and feel would be most useful to an improved understanding of the brain: philosophy.  I have had to surrender my hope, as have my colleagues; now the general public does as well.  We need to split our expectations in two.  We should all hope to understand the metaphysics of neuroscience – metaneuroscience, if you will.  And we should each seek to understand some, but not all, of the facts of the brain, and be prepared to explain them to one another.

Which is to say, we need not to get spun. Which is what surely happened to Lehrer.

Newspapers like the Journal need to examine their assumptions.  Yes, they can send a reporter to cover the White House and fight through the spin.  But who guaranteed them they can do the same with neuroscience? I honestly don’t think they can. And yet the solution can’t be to do away with solo practitioners like Lehrer.  It has to be the opposite: to make many, many more of them and link them together. Like neurons in the brain.

To close, my neurons just reminded me of some old Latin line about the government – ah, here it is on Google: quis costodiet ipsos custodes?  Who will guard the guards?

Well, consumers of neuroscience need to ask: who will read the readers? They can’t do it all by themselves. I think the answer must be all of us – and until then, more of us.  Otherwise there are just too many mistakes out there waiting to be made.  I have no doubt that even in this post I have made my share of them. Please let me know.

After all, I’m not a neuroscientist too.


About Peter Freed, M.D.

I am a psychiatrist (psychopharmacology and psychotherapy) specializing in the so-called "personality disorders," particularly narcissistic and borderline personality disorders. I was a Fellow and then an Assistant Professor of Clinical Psychiatry at Columbia from 2004- 2011, when I had an intellectual crisis-of-faith in which I stopped believing that neuroimaging could shed significant insight into the mystery of subjective experience. Since then I have focused on my clinical practice (at the Personality Studies Institute) and philosophical concerns.

View all posts by Peter Freed, M.D.

Subscribe to Neuroself.com

Subscribe to our RSS feed and social profiles to receive updates.

129 Comments on “Jonah Lehrer is Not a Neuroscientist”

  1. bevans Says:

    hi

    i think you confuse median with mean (or average) at a few points in your post.

    i dont think it changes the main point you make but it’s an important distinction, in general.

    bevans

    Reply

    • Peter Freed, M.D. Says:

      Well that’s the point – that I overlooked Lehrer’s use of the word median and, based simply on the gist of his prose, assumed he was talking about a group average. After all, he was writing about the wisdom of the crowd in the WSJ! I tried to identify for readers the moment that I realized he was confused about what median meant, but I think it may have been confusingly written! Thanks for reading, Peter

      Reply

      • kzizzle208 Says:

        You did a great job of explaining that moment you recognized the difference between median and mean. It was something that was nagging at me to begin with and I liked the way you set it up in the story. I’m no scientist so I put that thought aside because I thought at first that if I kept reading then the article would explain why Lehrer used median instead of mean (and why that was better). I really liked the way the writing followed your own thought process while investigating the “Blink moment” and I didn’t think it was confusingly written at all.

        Reply

    • Nurbek Says:

      , the fact that we’re “overdue” for a major U.S. landfall doesn’t mean there’s a geeatrr probability for one this summer because the probability of a major hurricanes making landfall from year to year in the U.S. in independent of the last major hurricane landfall.Wxskier…increases in SST’s have an affect on the formation of major hurricanes but has no effect on their path (i.e. whether or not they’ll make U.S. landfall). Also, we don’t have enough historical data to make the stochastic catalogs statistically significant in understanding the probability of hurricane landfall from year to year. Yes, catastrophe models used my insurers/reinsurers do increase the probability of major landfalls but that doesn’t mean they’re right. I mean, AccuWeather puts out a 15 day deterministic forecast but that doesn’t mean it’s correct, in fact in virtually never verifies better than simple climatology.Mark…I’m not throwing anything at anyone, I fellow meteo alum/friend of mine posted this on facebook. I was just pointing out that it’s silly to look at the graphs and make any conclusions.

      Reply

  2. Yisrael Says:

    Beautifully stated. As a PhD. student using EEG data to analyze lexical representation, I particularly resonate with your description of shitty fucking data. It is something that is so difficult to convey to a crowd that wants their wisdom in tweet length descriptions of findings.

    Reply

  3. forgottengenius Says:

    I am a new reader of this blog, but having made my way through the archives I have to say you write very beautifully and engaging. I’m looking forward to many more thought-provoking posts.

    Reply

  4. Bas Says:

    Excellent if somewhat verbose deconstruction of a particularly bad piece by Mr. Lehrer. What I don’t understand is all the preliminary reverence you seem to display before exposing this guy for what I already knew him to be just from following his Wired blog: a charlatan, with a charlatanesque feel for hype and zeitgeist, and an equally charlatanesque lack of substance.

    Reply

    • Peter Freed, M.D. Says:

      Thank you – and I will work on the verbiage! As to Lehrer – I honestly don’t know his work, only his media presence. I haven’t read his books or even his blog or heard him speak, but I have seen his name around a lot. So I can’t really judge the quality of his work yet. I am not sure I revere him, but I have a lot of respect for people who work hard, which he clearly does, and are trying to help us understand neuroscience, which we need to. It’s a tall order and I think a diversity of approaches is helpful – from the charlatanesque to whatever mine counts as. I am happy that he is writing, as he gives me and others something to think about & respond to. Someday I’ll read past the WSJ article! Thanks for reading!

      Reply

      • Bas Says:

        Thanks for your response. Neuroscience already being the hot topic it is today, it doesn’t need the kind of popularizing by fellow travellers like Mr. Lehrer, however hard he works at it. It needs thoughtful analysis of the far-reaching implications of its results, properly understood and critically examined. Which is why I’ll be following your blog.

        Regarding the Wisdom of Crowds: as you may know, the term itself appeared for the first time in 1907, as the title of a book by the British polymath Francis Galton. Following is the story of his first experience with the phenomenon, albeit written by another charlatan (yes, they’re everywhere), Derren Brown.

        “In 1906, Galton attended a farmers’ fair in Plymouth where he was intrigued by a weight guessing contest. The goal was to guess the weight of an ox when it was butchered and dressed. Around 800 people entered the contest and wrote their guesses on tickets. The person who guessed closest to the butchered weight of the ox won a prize.

        After the contest Galton took the tickets and ran a statistical analysis on them. He discovered that the average guess of all the entrants was remarkably close to the actual weight of the butchered ox. In fact it was under by only 1lb for an ox that weighed 1,198 lbs. This collective guess was not only better than the actual winner of the contest but also better than the guesses made by cattle experts at the fair. It seemed that democracy of thought could produce amazing results.

        However, to benefit from the wisdom of crowds several conditions must be in place. First each individual member of the crowd must have their own independent source of information. Second they must make individual decisions and not be swayed by the decisions of those around them. And third, there must be a mechanism in place that can collate these diverse opinions.”

        As we can see from the first two conditions, it seems Galton already noted the tendency of communication between subjects to actually lessen their aggregate ‘wisdom’. However, in extrapolating these results to the workings of democracy, Galton himself noted the following:

        “The average competitor was probably as well fitted for making a just estimate of the dressed weight of the ox, as an average voter is of judging the merits of most political issues on which he votes, and the variety among the voters to judge justly was probably much the same in either case.”

        This seems highly doubtful to me: we should expect most visitors to a farmer’s fair to be specialized (in varying degrees) in estimating the “net worth” of an ox.

        I guess we should prefer to state the result something like: the aggregate knowledge (or ‘estimation power’) of a group of (isolated) experts is higher than that of each of its individual members. Which is somewhat of a truism, really, and wholly inapplicable to any evaluation of the merits of democracy.

        Reply

      • link Says:

        Your posting really straightened me out. Thanks!

        Reply

  5. Trey Says:

    Dr. Freed,
    I’ve just recently found your blog, and added it to my reader. I found this article really helpful. I’m a grad student in neuroengineering at Drexel University. I’ve already been exposed to the rigorlessness of pop-sci, so while the whole article was interesting, I personally was most struck by your thoughts on the best practices for reading papers. I’m simultaneously learning, and being tested on, my ability to do this, and I can say that it isn’t easy. It had never occurred to me, for instance, to read the methods, tables and figures first. I will certainly do this from now on.

    If you take requests: your expanded thoughts on methods for digesting papers would be very interesting reading.

    Trey

    Trey

    Reply

    • Peter Freed, M.D. Says:

      Trey – that’s very encouraging. In a nutshell, I look at all figures first and their captions. Next I look at all the tables and get a feel for the numbers. Then I read the methods section well enough to understand the tables and figures. At that point I think I’ve absorbed 80% of the paper. Then its to results. Only if I am impressed with the paper at that point do I read intro and then conclusion, as this is where the spin is. Most papers I do not make it past methods because the study seems either wrong or trivial. I am sure there are other ways to do it. By the way, writing papers in the same order – figures, tables, methods, results, and only then intro and conclusion – is a good way to make sure your story follows your data and not the other way around. I’ll definitely get to your request at some point and see what people in other subfields recommend as well. Stay in touch, Peter

      Reply

      • Jon Pratlett Says:

        Thanks Peter. You really hooked me in and opened up my mind to many issues. I am engaged in post-gradeuate study in the neuroscience of leadership, and getting my head around how to effectively examine research findings, I find challenging and now even more essential. It is too easy to settle for the SFD. Thank you again and I am also interest, like Trey, in understanding effective means to tackle research papers more critically.

        Thanks also to the other contributors.

        Reply

        • Peter Freed, M.D. Says:

          Cool – give me feedback on what works and doesn’t, I am new to all this. Why don’t we all work on a post suggesting how to tackle research papers, divided by type eg cellular, fMRI etc. Start the ball rolling and I will chime in. This site need not be proprietary.

          Reply

    • mail order 50mg viagra Says:

      Super excited to see more of this kind of stuff online.

      Reply

  6. SBS Says:

    I am also new to your blog, but both you “blink” moment and the shitty fucking data hit home. I’m a relatively new special education teacher (4th year) n a field where shitty fucking data rules. I’m far from a statistician but I’m blinking so frequently in meetings I’m surprised no one thinks I’m a speed freak. While we should be learning more about neuroscience and brain development, instead we’re taking the shitty fucking data and its interpretation by education publishers at face value and foisting it upon our neediest student. Argh!
    Thanks for the blog. You’ve got a new reader.

    Reply

  7. sara Says:

    What an odd, length rant. Are you not familiar with the larger phenomenon of wisdom of crowds? Its been demonstrated in thousands of different studies and papers, starting with Galton. Lehrer is clearly using this example from the paper (which he accurately cited as far as i can tell) as a “for instance” of this phenomenon. The bulk of the article, of course, is a demonstration of precisely how it’s actually more complicated that is typically assumed, which seems to fit your main theme as well.

    Reply

    • Peter Freed, M.D. Says:

      I am familiar with the concept. But a) the single data point Lehrer chose was atypical – it was not a for instance, by any standards, and anyway was not from a category of data – median – identified as relevant by the article’s own authors and b) the data generated by the students were not consistent with the hypothesis. Besides that, it was a reasonable essay.

      Reply

    • Takehisa Says:

      Between me and my husband we’ve owned more MP3 paeylrs over the years than I can count, including Sansas, iRivers, iPods (classic & touch), the Ibiza Rhapsody, etc. But, the last few years I’ve settled down to one line of paeylrs. Why? Because I was happy to discover how well-designed and fun to use the underappreciated (and widely mocked) Zunes are.

      Reply

    • Grigory Says:

      16ad5352c1One thing I would really like to say is aawlys that car insurance canceling is a feared experience so if you’re doing the right things being a driver you may not get one. Many people do receive the notice that they’ve been officially dumped by their insurance company and several have to scramble to get further insurance after the cancellation. Affordable auto insurance rates are usually hard to get after the cancellation. Understanding the main reasons pertaining to auto insurance cancellations can help car owners prevent getting rid of in one of the most vital privileges available. Thanks for the suggestions shared by means of your blog. 151

      Reply

  8. PJorge Says:

    Great post. I am subscribing right now to your RSS.

    Best

    Reply

  9. Ian Sample Says:

    I’m afraid I didn’t make it to the second half of your post, but I wanted to make a point I suspect you might not have in the text I skipped.
    In Wisdom of Crowds studies you can look at the mean and / or the median. The median usually gives the best result if the guesses *do not* follow a normal distribution. The mean, of course, exploits the error-cancelling advantage that WOC is known for, that is, as many people under-estimate as over-estimate the right answer, so averaging cancels all but systematic biases. But to my point. To dismiss the median answer – one guy’s response – misses the fact that without the crowd you have no median answer to dismiss. Without the crowd, you do not know which value to pick. That’s the whole point. The crowd steers you to the median value, which in many cases outperforms the mean.

    Reply

    • Peter Freed, M.D. Says:

      Thanks for your feedback. I strictly follow the authors’ methods section. They do not advocate use of the median. Take a look at it, and then let me know if you think I’ve erred.

      Reply

  10. Ian Sample Says:

    My point is a general one about accepted wisdom of crowds methodologies. Averaging, or using the mean, is one strategy, but it doesn’t cope well with skewed distributions. In Galton’s original letter to Nature (Vox populi, March 7, 1907), he used the median (or “middlemost” value, in Galtonspeak) to arrive at his much celebrated answer for the weight of a “dressed” ox. The fact that this value was provided by one person doesn’t negate the power of the technique. Without the rest of the crowd’s responses, Galton would not have been able to identify a median weight.

    Reply

  11. Jonah Lehrer Says:

    Thanks for your thoughtful critique. I agree wholeheartedly that I’m not a neuroscientist and that complexity is good. That said, I fail to understand your issue with my citation of the data. I realize that, in this particular study, the geometric mean was a better demonstration of the WoC effect. That said, many other papers have found that the median does a better job, especially when dealing with crowds of naive, uninformed subjects. My real point in that paragraph, of course, was that the experimenters had repeatedly demonstrated the WoC effect, at least before they undermined it with social influence. You were clearly unimpressed with the messiness of their data. That’s fine. However, I concurred with the authors that their results showed that, under some lab conditions, the guesses of the group can come surprisingly close to the right answer. (Needless to say, this effect has been demonstrated countless times since Galton.) I agree that it’s interesting to delve into the details of geometric means versus medians versus arithmetic means as it concerns this effect. However, that’s not what this column was about, which is why I was unable to fully explore the issue. Instead, I wanted to give readers an idea of how the wisdom of crowds can actually work, which is why I cited the most impressive example from their paper. Such are the limitations of newspaper columns.

    Reply

    • Peter Freed, M.D. Says:

      Wow! I’m blown away to get a comment from you – thank you for reading this!

      That said, “the limitations of newspaper columns” is the problem in a big-picture way – what if that column wasn’t long enough to do justice to all you wanted to discuss?

      I am – I hope you don’t mind this – but I am probably going to do another post to explain why I didn’t like what in my lab we’d sweetly call your “cherry picking” of the “best” data point on the table to make your point. A “for example” should represent a random choice between all options. The data looks way worse when you count the whole median column, which the authors didn’t even want you to use. So in the end I felt that you analyzed their paper in the manner that one would analyze a text – interpreting it your own way – than the way I believe a science paper should be analyzed, which is really by either accepting-or-rejecting the author’s views. You can’t afford to fiddle around and use data they didn’t use to reach their conclusion better than they did. I hope to get to it later this week, but in broad overview for me the central concern is with how pop neuroscience is conducted. In the case of your article, the issue turns on the difference between how science views a crowd and how the public does – I think something is lost in translation when you use the median. I’ll save my idea for my post. Also people want to know – why didn’t they put in a link to the Swiss study?
      Thanks again for your gracious comment, and I hope we stay in touch. Peter

      Reply

      • Jonah Says:

        Hi Peter, Thanks for your reply. In general, when writing a column (700 words!) one has to prioritize depth of analysis. In the paragraph of the column that you’ve focused on (and spent a few thousands of words analyzing), I wanted to make a simple point, which is that the wisdom of crowds effect exists and was confirmed (once again) by this paper. If that’s all the scientists did, or if that’s what my column was about, or if that was the focus of the actual experiment, then I would have spent more time analyzing their data and mentioned their use of the geometric mean as opposed to the more conventional median. I also would have been sure to mention that the one example I cited was the best example of the effect. But, of course, that wasn’t the point of my column or the paper. And I politely reject your notion that I reached a “better [ie, stronger] conclusion” than they did concerning the wisdom of crowds effect. As you’ll see in their paper, they spend very little time (as did I) discussing their confirmation of the wisdom of crowds effect, precisely because it’s such a well-demonstrated effect. Given the limitations of the column format – and it is a limited format which I certainly struggle with – I felt it was more important to save my words for the real substance, purpose and point of the paper, which is that the wisdom of crowds effect can be undermined via social influence. I also believe that there are many different ways to read a science paper, and that your method of rejection-or-acceptance is not the only valid way. Sometimes, it’s important to place an interesting new result in a broader context, which is what I was trying to do in the column.

        Lastly, I’m afraid I don’t control or dictate the editorial policies of the WSJ re: linking. I always include a link to the paper in my blog.

        Reply

        • Nettie Fisher Says:

          It is unfortunate that you put forth these observations about Lehrer’s work without giving it more careful consideration. All of your commentary, while valid, overlooks a main point of his first book, and the reason why I am drawn to his work in the first place.

          Our realities, and the scientific theories we come up with to relatively quantify the things in them, are determined subjectively by the individual.

          Which implies that the truths you employ to fill your reality are not the reflection of truths that exist outside yourself as laws of science or nature (observable as you suggest in data and scientific observation) but are in actuality not any sort of truth at all. They are merely your interpretation of some recurring observations, given meaning and importance only by your attention to them. “Scientific facts are meaningful precisely because they are ephemeral, because a new observation, a more honest observation, can always alter them.”

          Your tone at the end of the article reflects this implication, but you take the stance that as a neuroscientist you have more credibility and therefore a more accurate understanding of this study, that your method for decoding scientific data will reveal a “truer” perception of reality. But, even your method for tackling this information, your suggestion to get an overall feel for the data before trying to decode it, shows your understanding of the frivolity of trying to dissect every detail. Lehrer picked median over mean because he knew the importance lies not in finding answers in specific, quantifiable “facts”, but that the truth is formed through opinion and interpretation.

          I am glad Lehrer is not a neuroscientist. Neuroscience poses many questions, but Lehrer has realized there are no answers. He knows the answers are different and constantly changing for everyone, and that makes these questions unanswerable. Lehrer started looking for questions and stopped looking for answers because he knows that “we solve ourselves.”

          I strongly encourage you to read his first book. It would be impossible to fully convey his ideas through a comment on a blog, which is why I’m sure he didn’t bother to try, but that’s why he wrote a book. You renounce his ideas because he arrived at a different answer, but your thinking parallels his. Any answer has the potential for truth, Lehrer’s just trying to ask questions.

          Reply

          • ...sleeprunning... Says:

            Here’s a way for JL to make amends — do a paper (more than 700 words please) on how a general reader can understand stats in science reporting.

            JL can make it a task to educate his vast audience on better understanding, not necessarily the original research papers, but critical reading and thinking of stats in pop sci.

            This basic understanding is a critical skill for the educated person today. JL is clearly skilled in reaching editors, producers and readers so he can probably make the topic not completely boring and a turn-off.

            It can be a series of articles in fact.

            Reply

  12. Sandeep Gautam Says:

    I would like to focus on a few things:
    1) Median may be as good an interpretation of a set of data as mean (arithmetic or geometric) or mode may be. Median is especially relevant when there may be outliers in a data set which skew the normal representations of data like mean or mode. Median though consisting of a guess of a single person, has to be seen in context of represnetativeness of groups guesses so is as much a group aggregation measure as is mean/ mode.
    2) what about the data set. Does the data set has a typical Gaussian distribution where mean , mode, median are same. Consider New immigrants to a city either varying by city or by year; if the distribution is more like students T tailed, perhaps if the guessing function was also approximated as a t tailed distribution and Median chosen as a representative average value then perhaps the wisdom of crowds is indeed true and median is what we should bother with. If however population densities of diff cities vary as normal distribution perhaps a mean approximation of guesses is more appropriate. The point I am trying to make is that mean may not always be the best measure and when we club population densities and immigrants together in the same data set we might be comparing apples to oranges if u=we use the same average measure (mean /GM/AM/Mode/Median whatever) .
    3)In light of above one needs to analyse and treat each data set independently and see whether wisdom of crowd effect is real or not.

    Reply

  13. Jennifer Ouellette Says:

    This is a truly excellent deconstruction of how the best intentions can go horribly wrong. And speaking as a science writer — who is NOT a physicist yet writes about physics, and must continually ask people not to elevate me to “expert” level :) — I appreciate that you made a thoughtful, thorough critique without attacking Jonah Lehrer personally (unlike a couple of your commenters). And you refrained from lapsing into the usual, “That’s why only scientists should write about science!” fallacy that usually emerges from such discussions. The truth is, whatever one’s opinions about Lehrer and his work, we do need popularizers of science… and we also need scientists like yourself providing useful critiques to help make sense of all the noise and spin, particularly in a complicated area like the brain. Those are two different, equally valuable roles and I wish more scientists would realize this.

    Am definitely adding your blog to my feed. :)

    Reply

    • Peter Freed, M.D. Says:

      Thank you! Yes, I really HATE the impulse to run people out of town, especially people like Lehrer who are clearly improving the discussion, dissemination and debate, and highly motivated. We need to all pitch in. We aren’t waiting for Jesus – we’re trying to get the public more comfortable thinking and talking about the brain. And the sooner everyone is comfortable with the methods sections of papers, the sooner scientists will stop spinning so much. I still don’t feel I understand the paper fully, especially after some of these very interesting comments. Peter

      Reply

  14. Eric Michael Johnson Says:

    Very nicely done. Perhaps the very reason he chose not to cite the paper and authors as he usually does.

    Reply

  15. Katie Says:

    DISCOVER magazine, not Discovery

    Reply

  16. The Main Gauche of Mild Reason Says:

    To be fair to Lehrer, I think this is more of a “peer-review fail” than anything else. For the reviewers to accept the use of the geometric mean (a device usually used in determining the central tendency of percentages or other numbers that multiply rather than add) as a metric for the central tendency of primary numbers is pretty bizarre.

    To those who continue to claim that the paper still shows something useful, I would call attention to the fact that these numbers only vary within an order of magnitude or so (hundreds to thousands), and that by mathematical necessity, the geometric mean of a set of always much smaller than the arithmetic mean; in this case it merely counteracts the tendency of the people in the study to (sometimes vastly) overstate numbers. Getting the right order of magnitude isn’t really that remarkable.

    Also, I would question the choice of metrics used by the authors—ideally, you’d want examples that a) vary over several orders of magnitude so are harder to guess and b) that guessers are likely to UNDERESTIMATE as well as overestimate (there are no underestimated examples here). It’s more a poorly designed study than anything.

    Reply

    • Peter Freed, M.D. Says:

      I wish we could do a group post by all the people who chimed in about this issue. Maybe I will try to organize something by email. I don’t understand this issue as well as I wish I did – we should also discuss the difference between standard deviation and standard error, since everyone with bad SDs always puts SEs on their bar graphs – if they put any bars at all! ;-) I hope its okay if I email you at some point about such a post. Peter

      Reply

      • Ryan Says:

        Your point about SDs and SEs (SDMs) is very important. That would be a great blog topic. When tempted to use SEs as the main component of uncertainty, I ask myself this: If I took a million data points, would be total uncertainty in the quantity I am reporting really be reduced by 1/1000? That’s rarely the case, even for a simple measurement. For instance, try measuring the length of a table with a ruler to 1 mm precision. Now repeat 100 times. Do you now really know the length to 100 micrometers? No, because at that precision other uncertainties, that could previously be ignored, become important.

        Reply

    • BubbaRich Says:

      It does NOT counteract a “tendency” to overstate numbers. It tends to moderate outliers. A “tendency” to overstate numbers will remain. You need lower guesses in the mix to bring the geometric mean down, which eliminates your “tendency.”

      Reply

  17. Steve Says:

    A great and entertaining post. Glad I found it and your blog via an FB post.

    Reply

  18. Jordan Suchow Says:

    Following up on Ian Sample’s point, Francis Galton wrote a letter in the 28 Feb 1907 issue of Nature arguing that the median is the fairest way to aggregate group responses. The mean is a poor choice because it allows rogue voters enormous impact: a single voter can sway the outcome by choosing an absurdly large value in the desired direction. Choosing the middlemost value (the median) gives each respondent one vote. The aggregated outcome is then the value for which half of the people would say “too high” and half “too low”.

    Reply

  19. Owen Ferguson Says:

    Really interesting post and discussion. I’m glad I stumbled across it.

    I’m not a professional scientist, but I do have a question about the table data and your interpretation to help me make sure I’m fully understanding the paper.

    The caption under the table reads:

    “The aggregate measures arithmetic mean, geometric mean, and median are computed on the set of all first estimates regardless of the information condition.”

    I read that to mean that the table isn’t reporting data for measuring the ‘wisdom of crowds’ effect, because the table is reporting based on data where some of the participants were under the “aggregated information” condition, while others were under the “full information” treatment.

    So, it looks to me like the table isn’t that great a source for identifying the ‘untainted’ wisdom of the crowd which you;d expect to be simply without the “aggregated information” or “full information” conditions. (Frankly, the table seems to be pretty useless full stop, but that may be because I’m just not getting it).

    The really useful data appears to be in the figures and figure B in particular since that provides a ‘wisdom of crowds indicator’ for each of the three conditions which seems a reasonable enough measure to me, with my basic stats understanding.

    So, is the table necessarily the best source to identify whether or not the crowd is ‘wise’ or not in the first place? After all, it looks to me like it contains the guesses from the ‘social’ conditions that the authors suggest adversely influence the crowds accuracy.

    Hope someone can con firm whether I’m getting it or just confused.

    Reply

    • Peter Freed, M.D. Says:

      Hey thank you – I only focused on Table 1. Analyzing the whole paper was way too much to ask of myself on a vacation weekend, especially as Table 1 was what my concern was with.

      Reply

  20. cathy kerr Says:

    Given that analysis of the Median is a legitimate strategy — it controls for when people lob “wacky” guesses (eg., 200,000 immigrants) that skew the mean…. It seems a little pedantic to focus on this point….

    Instead, what strikes me is that median values do not look particularly “wise” except in the one instance (the immigrant item)… So wisdom of crowds does feel like a shifty phenomenon that is not validated by the paper in question …. It seems like your quarrel is with the reviewers of the PNAS paper, and only secondarily with Lehrer.

    Also, what do you make of the Swiss authors’ assertion that wisdom declines with social influence?

    Reply

    • Peter Freed, M.D. Says:

      I don’t think I have a quarrel with anyone. I think Lehrer cherry-picked his median value to a single question to make a rhetorical point, but it overplayed the significance of the WOC in the reader’s mind – the reader who could not see the original citation.

      Reply

  21. Matt Robinson-Plume Says:

    Superb post

    Reply

  22. Elmer Rich Says:

    “All science is wrong, but some is less wrong.” Like the post a lot. Any form of human knowledge is more wrong than right, grossly imperfect and conditional. There is some evidence that group decision making is superior.

    An interesting alternative is covered in J. Lehrer posted on this, here: The Reason We Reason — but we beat him to it “Using Reasoning to Persuade: Making Sense of “Irrationality” + How Reasoning Helps Groups But Also Drives Bad Behavior” — http://wp.me/p167Bf-8x

    Here’s an excerpt: http://wp.me/pXvvI-6f

    Brain science is just newborn, not yet even crawling and still “a booming, buzzing confusion.” It has a face and pop validity which is a strength and serious flaw. Many neuro commentators are young and inexperienced and get paid, like all journalists to:
    – Support the status quo of advertisers
    – Hyper-seek supposed threats to that staus quo

    Of course, there is always David Brooks – ugh. We probably need more women writing about neuro stuff, e.g., lots more humility.

    Of course none of us have any idea of what we don’t know. No one is going to pay a journalist for ever writing about that.

    Maybe there will be journalists paid to challenge the status quo someday but likely not.

    Reply

  23. BubbaRich Says:

    Yes, I think you’ve missed it by a little bit, based on that paper, which goes along with what I know of central tendency statistics. In fact, the authors don’t explicitly say WHY they didn’t use the median as their metric. They do say that the idea of the median validates their use of the geometric mean as a metric, because it is close to the median.

    Your criticism of Lehrer is spot-on, he abused the English language and reader’s trust by using an atypical result as a “for instance.” But the median IS a good measure of a group central tendency, and THAT doesn’t undermine this work in any way. Both that and the geometric mean tend to reduce the effect of outliers in the data.

    Reply

    • Peter Freed, M.D. Says:

      Thank you – yes, they never say why they don’t use median – but they don’t, and my larger issue was that reporters should not report statistics that aren’t used to reach the conclusions of the paper.

      Reply

  24. Melanie T Says:

    I just want to point out that for whatever the original article is or isn’t (and I will admit to not having read the articles involved), taking issue with the median because it is technically a single person’s guess is not really relevant to whether or not it could potentially be used as an indicator of the group. It’s a single guess, but it’s also a measure of central tendency, just as the mean is. Medians and log transformations are *frequently* used when there’s a skewed distribution – you seem to take issue with the fact that the original authors log-transform their data as well (in the geometric mean), but this is fairly common (at least in social psychology) if one doesn’t want to simply exclude extreme outliers entirely.

    Looking at the data table, it seems odd that the arithmetic mean for assaults (for example) would be approximately 135,000, while the median is only 4,000. This means there must be some EXTREME outliers at the very, very top of the spectrum throwing everything off. In this scenario, the *only* way to get an accurate picture of central tendency would be to exclude the outliers entirely, log-transform data, or use the median as the measure of central tendency. Using the arithmetic mean, which is clearly greatly swayed by extreme datapoints, would actually (in my opinion, and the opinion of many others) be unethical.

    Again, I’m not trying to speak to the data interpretation itself – just the idea of using a median or a geometric mean to measure central tendency. In a distribution that’s as skewed as this one clearly was, it’s common practice.

    Reply

  25. Giovanni Santostasi Says:

    While I applaud you for the investigative work you did in finding the silly and misleading mistake of Mr.Lehrer, not understanding what is the significance of a median, I don’t agree with your general conclusion. Let me start with saying that the questions asked to the students mentioned in the paper are a kind of problem (making an educated guess) called Fermi’s problem. From Wikipedia :”In science, particularly in physics or engineering education, a Fermi problem, Fermi question, or Fermi estimate is an estimation problem designed to teach dimensional analysis, approximation, and the importance of clearly identifying one’s assumptions.” While the Swiss students didn’t go consciously through the steps of a typical Fermi’s problem (I imagine they had to give quick intuitive answers) unconsciously they maybe have done exactly that. But even if they had the time and inclination to go through the steps, a typical Fermi’s problem allows for an error in the estimate that is up to an order of magnitude. In other words, if the exact number is 10.000, guessing 30,40, or even 90 thousands would have been good in the context of a Fermi problem. So getting an error in the hundred is actually pretty good (it means a factor of fews that is basically nothing). Fermi problems are considered a golden standard in the context of a guessestimate and in fact it takes a good deal of convincing to explain students that guessing wrong by a factor of 10 as a first rough estimate of the numerical answer of a problem, where little or no information is given is actually pretty good and a useful thing to do.
    So actually the single individuals have done pretty good in most of the cases in the experiment (with some exceptions as in the assault estimate).
    Second, there is a reason the geometric means is actually doing so much better.
    Again from the definition of Geometric Mean in Wikipedia:
    “Although the geometric mean has been relatively rare in computing social statistics, in 2010 the United Nations Human Development Index switched to this mode of calculation, on the grounds that it better reflected the non-substitutable nature of the statistics being compiled and compared:
    The geometric mean reduces the level of substitutability between dimensions [being compared] and at the same time ensures that a 1 percent decline in say life expectancy at birth has the same impact on the HDI as a 1 percent decline in education or income. Thus, as a basis for comparisons of achievements, this method is also more respectful of the intrinsic differences across the dimensions than a simple average.”

    The idea here is that if you have statistically independent samples, representing different processes and parameters of a problem, taking an arithmetic mean is not that significant. A geometrical mean characterizes in a more meaningful way the different weights of the individual estimates. An intuitive understanding of how this could be useful in the context of crowd wisdom is to think about where the source of crowd wisdom may actually be (if it exists at all). Imagine that in a small group of students that don’t have any clue about how many immigrants there are in Zurich there is one or even few that are immigrants, or have studied the topic at school, or have read for themselves some material about the subject. Their answer will be much closer to the correct one. An arithmetic average would wipe this “higher wisdom” if the number of expert is small (as indeed is the case for random crowds). Statistically the answer of the non experts would be quite all over the place while the response of the experts would be in a narrow range. You can consider these two types of answers as different “dimension” and they should be emphasized in a different way that is what the geometric mean does. It is actually a significant result that the Swiss scientists were able to show that the geometric mean works much better in this case. Now a more relevant and interesting question is how then there is anecdotal evidence that we recognize this “geometrical mean wisdom” in crowds? Do we do a geometrical mean calculation in our heads when we ask the crowd for a word of wisdom on a particular question? Well, I think it has to do on how the data is displayed. Let’s take an example, that is the popular game “Who wants to be a millionaire?”. One of the life savers is to ask the public to help, in other words to use the wisdom of a crowd. I noticed, that rarely the crowds are wrong. How the data is displayed in this particular example? In a histogram, showing the different counts for the possible answers. If most people don’t have a clue the distribution would be flat. And in fact, probably, for the difficult questions the distribution would look pretty flat. But if in the crowd there are few experts the right answer would stick out of the flat distribution as sore thumb. Geometrical means, and our eyes looking at a distribution are good in picking up this spiky features. An arithmetical mean would be pretty useless in this context. If I’m right in this analysis, then this should also explain why communicating crowds would destroy the wisdom effect. Most people would listen to what the majority has to say and the experts would not be listened and in fact they too, being humans, may feel compelled to change their initial guess (in particular if they were not truly experts and they were simply just better than the other ones in making educated guesses). I think this idea of the wisdom of crowds deserves more investigation and in particular the usefulness of the geometric mean or other statistical ways of the extracting the wisdom from the crowd should be explored. Indeed, what distinguish a scientist, professional or citizen, is patience, tenacity and taking the time to understand both the general picture than the details of a problem.

    Reply

  26. Robert Says:

    To be Devil’s Advocate for a moment, the median values you discard as being not a “group” answer could not be determined without a cohort, being found in relation to the other members of the cohort. No group: no median. And though the median estimates in the study are not generally accurate or even that close, they are not absurdly off-base either, compared to random numbers chosen from a vast range of possible answers.

    So while the paper doesn’t prove that groups are smart, it does suggest that an answer closer to truth might be found in a group of responses, closer to the median than to an average. And that if you need to test an answer for “sanity”, you might do worse than to compare it to a median answer coming out of a group; you could not do that if you didn’t have a group of answers to draw from.

    But enough with being the Devil’s advocate. Society doesn’t make too many “decisions” per se, certainly not based on median answers elicited from disinterested individuals by simple questions about numerical facts, so there really isn’t much practical comfort to be found there.

    Reply

  27. ElOceanografo Says:

    Good post–I especially liked your point about freaking out after looking at the table for three seconds. But to pick a statistical bone, Lorenz et al.’s use of the geometric mean and median here is actually pretty well justified. Looking at their data (available as an XLS here), the estimates are all highly skewed. They’re more or less log-normal with a few extreme outliers on the high end, meaning the arithmetic mean will be affected disproportionately by the high values.

    If the crowd’s estimates are predictably skewed, then we’d want to use the particular aggregate statistic that falls closest to the true values. If a particular crowd wayyyy overestimates some question, the “wise” statistic could end up being, say, the 12.7th percentile or something.

    Reply

  28. Ryan Says:

    Your main point, that Lehrer used the extreme value as his “for instance”, is important. Journalists often use extreme cases, but at least they usually prepend it with “up to” or “fewer then”. Calling this just an example, is definitely misleading.

    Beyond that, Lehrer’s use of “median” is unambiguous. I realize that you misread that as “mean” due to expectations (maybe your crowd, like mine, prefers weighted means), and it is interesting to read about how your expectations influenced your perception, but you seem to be accusing Lehrer of more than his crime. The median is one of many estimates of a central value of a distribution, along with the arithmetic mean, geometric mean and others. That is, if crowds have means, they also have medians. That median may or may not be equal to the guess of one person, depending on whether the the number of guessers is odd or even. (In this case, 23 people guessed exactly 10,000). The median is one measure of the central value of a distribution. In many studies, the median is the most robust average, since it is less distribution-dependent than the various means. It is a good statistic to report in a non-technical piece since it is so robust. In the case of the PNAS article, the authors did a nice job of calculating 3 averages (arithmetic mean, geometric mean, median). They noticed that the median and geometric mean both worked well, but preferred the geometric mean since their data was manifestly, and explicably, log-normal. They conclude: “Notice, that a high wisdom-of-crowd indicator implies that the truth is close to the median. Thus, it implicitly defines the median as the appropriate measure of aggregation. In our empirical case this is not in conflict with the choice of the geometric mean as can be seen by the similarity of the geometric mean and the median in Table 1. A theoretical reason is that the geometric mean and the median coincide for a log-normal distribution.”

    In conclusion, I wish more journalists would report medians instead of extreme values when summarizing research. Medians are more robust than means, at least if uncertainties are not reported. An appropriate mean with uncertainty would be even more informative, when available.

    Reply

    • Peter Freed, M.D. Says:

      You and many other people have made wonderful points about the proper use of statistics, and I hope to respond to them and to incorporate everyone’s thoughts into my post. I’m slightly awed by all the energy people have put into responding to what I wrote, and so I have just approved everyone’s comments to get them up but am taking a bit of a breather. I hope to respond soon. I wish there was a way for everyone who has commented to talk to one another, as I see myself as starting the conversation but not owning it. Is there such a thing out there? A sort of twitter for long conversations? Peter

      Reply

  29. Marjorie Says:

    I am also Not a Neuroscientist. No, really – I’m REALLY not a neuroscientist. I am one of those smart, well-intentioned, curious, avid readers who needs to spend more time chewing before they swallow. THANK YOU for reminding me to blink more often.

    Reply

  30. Medea Says:

    Loved your essay-length and all. My son is an engineering major/philosophy minor at UC Davis and I sent it to him-he’ll be your next big fan (after his sainted mother, of course.) I was drawn to this piece because I have read Lehrer and was captivated by the way he made science comprehensible to a liberal arts person like me–now I find a lot of that is because he doesn’t really get the data either, which makes me feel better, just more confounded about the subject. You’re right-it will take a lot of people reading voraciously, policing the writers and researchers, and putting correct information into the minds of those interested to know what really makes our brains work, and what makes us ‘us’. I have a friend who just had brain surgery, so this is a burning issue for me right now–how much of ‘him’ will be lost? How much could he have had to lose had the scalpel slipped? And really, what about those conjoined twin girls in Vancouver? Thank you for a wonderful and thought-provoking essay!

    Reply

  31. marcoiac Says:

    not sure if somebody else said this, the blog is already quite long (i haven’t read it all, and i read zero comments), but anyway, here’s my two cents to what seems such a crucial statistical issue and is not (unless, again, i am not getting what the issue is, since i haven’t read the whole thing).

    medians are better than means in that they are less sensitive to outliers. still, they represent a population, even if they are just one value. medians do not exist as isolated entities

    Reply

  32. MRW Says:

    Three comments:

    1) I’m not so sure Lehrer was confused about what the median is, but I’m having trouble believing that *you* are clear about it. Medians are not guesses by a single person by any reasonable understanding.

    Lets say that we had the following guesses for some number:

    9, 10, 10, 11, 1000.

    What number owes more to a single person’s guess, the arithmetic mean of 206 or the median of 10?

    2) 38% off doesn’t seem so bad for guessing quantities like this. The real point is that very few individuals would have an average error as low as the group’s, so you are better off going with the group’s geometric mean or median response than going with the guess of one person.

    3) The choice of median over arithmetic mean is a good one. The choice of median over geometric mean is justifiable given the nature of the WSJ article and the fact that a geometric mean and a median are generally very close. However, the choice of the one with 0.7% error was clearly cherry-picking. There’s no excuse for it.

    Reply

    • Peter Freed, M.D. Says:

      Thank you for your thoughtfulness. However my critique had to do with his use of the 0.7% in the context of his article, not in the context of statistical practices within science. I’ll try to explain in the next post – and I agree that I did not explain medians very well. They represent the guesses of either one or two people “nominated by the crowd.” Such nominations are not what the general public thinks of when they think of crowds, which is my concern.

      Reply

      • MRW Says:

        The median isn’t the guess of “one or two” people. It’s often a better representation of the overall guesses of the group because it depends less, not more, on what any one person guesses.

        I agree that the 0.7% was a bad and misleading number to use (and already said as much), but not because it was the median. It was a bad number to use because it didn’t accurately reflect what the medians tended to be in these group guesses.

        Reply

  33. Anonymous Coward Says:

    To the OP:

    I admire your candor in admitting your own poor understanding of mathematics, it made for an entertaining read.

    But I think you’re too hard on Lehrer. His language about the median was straightforward. After I read “the median guess of the students was 10,000,” my first reaction was something like: “That’s not surprising; of course the median is likely to be a round number. Most people have a strong bias for answering in round numbers.”

    But for heaven’s sake, after realizing your error, surely you should try to understand what a median is before saying ridiculous things like “median guesses are not guesses by a crowd”.

    Unlike Jonah Lehrer, the original poster may indeed be a neuroscientist. But for a scientist, he or she is borderline innumerate.

    I’d recommend Chad Orzel’s blog discussion of this same issue for a more in-depth take.

    Reply

    • Peter Freed, M.D. Says:

      Hey! Thank you for this – and for the recommendation of Orzel’s blog. I promise to keep up the candor even if I continue to look like a dumbass. I will be trying to “explain myself” in the next post. In it you may be amused to know that I plan on digging my hole a bit deeper – I do indeed believe that for the general public, median guesses are not guesses by a crowd – particularly when the general public is not told that the reason the median is being used because the distribution is right skewed, and the skew is not given, not to mention the cherry picking. For the general public, a median guess is what is known as “representative democracy” and is, I think for many of them, quite the opposite of a crowd.

      Reply

      • Anonymous Coward Says:

        Perhaps I should wait for your next post, but I don’t understand why median isn’t a good descriptor for the crowd.

        5 years ago in my town, the median price for the sale of a house was $350k. Now the median price for the sale of a house is $170k. I think that tells you a lot about the sales price of houses in my town.

        I completely agree with your comment about the distribution about the median being “right skewed” (questions about wealth distribution in the U.S. notwithstanding). But if you just take an arithmetic average, your answer will be disproportionately controlled by the values that the “high guessers” are giving: those who guess high can have much more influence over the arithmetic average than those who guess low. Taking the median effectively gives more equal weight to all the guesses, which I guess is what you mean when you refer to it as “democratic”?

        Similarly, given the typically logarithmic distribution of numbers (see Benford’s law), a geometric mean is often a more appropriate descriptor of a distribution than an arithmetic mean.

        Reply

  34. Dussault Says:

    “That thing [the brain] is exponentially harder to understand than any other phenomenon in science – some people say the universe.”

    The complex system that is the brain is part of a greater system that is the universe, so of course the universe is more difficult to understand than the brain (not that anyone is about to fully understand either).

    To enter the world of sci-fi: the limited nature of our brains also poses a limit on our capacity to understand the universe, but if we manage to understand our brains enough so as to raise this ceiling, then we will both raising the ceiling in terms of our ability to further raise the ceiling, and further understand the universe. This may trigger a cycle of self-modification which effectively eliminates this otherwise inherent limit to our understanding of the universe. How far can our understanding of the universe go? This is a question that I cannot answer, but only contemplate.

    Reply

  35. .....sleeprunning..... Says:

    Couple of thoughts.

    Stats — Maybe it would be productive to discuss the distribution of data points as a context for measures of central tendency. We can hyper-focus on the those and forget the richness lies in the data distribution itself. For example, we have not discussed non-parametric tactics. These can be quite robust.

    Communicating Science — It seems like there are (at least) three levels of science communications: Professional, Public and Popular.

    Likely what we see represented by JL is the best we are going to get with Popular Science communications. It’s not perfect and frequently wrong but it does serve the purpose of widening the audience, credibility and appeal of science — that seems a plus. JL’s main flaw seems he is young and inexperienced — but cute (it is frequently said). He hasn’t yet put in the years it takes to develop a mature voice and critical sense. He will.

    But likely most journalists are going to be inexperienced nowadays.

    Public Science is what we seem to be discussing here. Public science has pretty high standards. It is not for a mass audience but is a yeoman attempt to translates professional science into meaningful and usable knowledge for problem-solving. Our sense is that Public Science is mainly targeting other professionals and the application of science by non-scientists or even scientists in a different discipline. We’ll think about this some more.

    Reply

    • Peter Freed, M.D. Says:

      Amen to your first point. Why not report mean, median and skewness? However this is far too much for the WSJ. I guess the question is how to describe such things to the common reader – my view is that you report the variable REPORTED IN THE PAPER. I simply don’t think pop science can function when reporters report data that was not used to reach the conclusions being discussed without acknowledging it.

      Reply

      • ...sleeprunning... Says:

        Indeed, but the problem (is really editors and producers who really decide) is that what generates the most attention is:
        – Fear
        – Supporting the status quo and ideologies of the advertisers first, the readers

        So we ain’t going to get anything that violates those requirements. However, does the vastly broader awareness that pop sci generates balance out the misrepresentations and misunderstandings? That’s probably not even an empirical question.

        Maybe here’s an example: If I am selling “real” neuroscience, it helps me to use JL and WSJ as “hooks” for instant credibility — regardless of what their content is. Both are “brands” so I (shamelessly) steal theri brand credibility to halo effect my ideas. Sneaky, huh?

        Reply

        • Nazareno Says:

          I\’ll gear this review to 2 types of pelpoe: current Zune owners who are considering an upgrade, and pelpoe trying to decide between a Zune and an iPod. (There are other players worth considering out there, like the Sony Walkman X, but I hope this gives you enough info to make an informed decision of the Zune vs players other than the iPod line as well.)

          Reply

          • Wylder Says:

            16ab435142Hello there! Great article! I am a usual viiotsr (even more like addict ) of this website sadly I had a is sue. Ie28099m just not really certain if its the right site to question, but you have no spam comments. I get comments day by day. Can you assist me? Thanks for the tips! 151

            Reply

            • Amira Says:

              again though, then why the re-emergence of hosunig? even during the popping of the bubble?my initial thought (just a though, so please keep the comments coming) is it is a reflection of leverage. during the 1980’s leverage in corportations (equities are just levered portions of a companies total value) was rewarded as financing became cheaper (interest rates dropped dramatically), but individuals didn’t think of homes as investments yet (i.e. they just lived there and didn’t use home equity to buy another home / buy a new car).since 2000 homes became the leverage vehicle of choice due to no money down deals and the like.now, leverage is coming crashing down, but it turns out equities (and consumption by individuals) was a levered vehicle based on the value of homes themselves, which individuals used as an ATM to support consumption, which supported equities. The reason equities haven’t rallied is since 2000 they didn’t invest in themselves, they just paid back owners via dividend or stock buyback.no ATM = no consumption = a fall in equities more than hosunig, which at least can still provide shelter

              Reply

            • Marlee Says:

              http://yourmedicalinfo.net/|lasix bumex :((( http://findcarinsurquote.com/|insurance vehicle ratings exzucx

              Reply

            • sildenafil Says:

              I searched a bunch of sites and this was the best.

              Reply

            • Kim Says:

              I had a rather hard time cnoosihg just one type of physician I would want to work for. So many of them fascinate me, and with me not really going into any medical field other than support, I never gave this any thought in the past. After reading the list, I am more favorable of working for a neonatologist. It is difficult to think about how neonatologist physicians sometimes have the most difficult job in the world, but I can only imagine how amazing it would be to be a part of saving a baby’s life. I had a coworker once whose baby was born at 36 weeks, and her baby had a lot of heart and lung problems. There were concerns about whether or not they would ever fully develop once she had him, but after many months in the NICU, and many scares that happened during it, the doctors were able to save him and he is now a very healthy 5 year old. It is because of that I have a higher interest in the neonatologist field.I hate to say which type of physician I would care less to work for, and it is because I worry that many will take it the wrong way. When I was 16, I used to help my mom at an assisted living home as a caregiver. We would get to work at 7:00 A.M. every morning to prepare breakfast for four of the elderly men and women that we were caring for. We would then make sure that all bedding was changed, rooms were cleaned, meals were prepared, and appointments were handled. We worked 12 hour days, and they were always grueling. The owner of the home made sure that everyone had their medicine and made it to their doctor appointments on time. However, she was more worried about getting paid for her services than actually helping the elderly. She would yell at them if they did something wrong, and even call them terrible names. My mom reported her and we both quit our job, but it has always left a sting in my heart since then. It is because of my experience with that situation that I do not think I could ever work for a gerontologist. I know that the situations would be much different, but ever since my experience with caring for elderly individuals it is very hard for me to think about assisting a physician in geriatrics because I worry that someone else might treat the elderly in the same way the owner of the home did. I am a firm believer that the elderly deserve the ultimate care and comfort when going through any treatment and aging in general, but I do not think I could ever work in that environment again.

              Reply

            • http://www.mamasgotwanderlust.com/wordpress/china/objective-and-up-to-date-for-planning-a-tibet-tour.html Says:

              Your articles are for when it absolutely, positively, needs to be understood overnight.

              Reply

  36. Minivan Says:

    This is a great blog! Will follow in my RSS feed. However, it might be best to cut down posts to no more than 500 words. Just a suggestion.

    Reply

    • Peter Freed, M.D. Says:

      NEVER! Goodness, what is the point of finding a platform on which to pontificate if one does not then, in fact, pontificate? You underestimate the cathartic effect of typing. Nevertheless, I will try to be more concise.

      Reminds me of my favorite online dating biography every: a female friend of mine showed me a guy who had written her. The question he was addressing was “describe yourself in 500 words or less”.

      He answered: “concise.”

      I told her she absolutely had to date him.

      Reply

  37. neuromusic Says:

    Nice critique of this table and Lehrer’s use of it.

    I think that the choice of a measure of central tendency is an important one. It is the statistical equivalent to the Wisdom of Crowds effect… a reduction of the data to one measure. Statistically, WoC is somewhat trivial: of course pooling data from multiple sources (whether they be individuals or trials or studies) will give be a better estimate than one source. What we see here are multiple disagreements on how and where to do this.

    I like your “median as representative democracy” characterization… apparently, our society as institutionalized different statistical methodologies in sampling the Crowd (not to mention choosing which part of the Crowd to sample from). American democracy, the stock market, and American Idol each take different statistical approaches to determining what exactly it is that the Crowd thinks.

    I think that this underlies part of people’s criticisms of your criticism of Lehrer using the “median”… the appropriate way to Sample the Crowd is not clear and whether Means or Modes or Medians are appropriate or whether they should be Arithmetic or Geometric or whether the second or third or fourth moments ought to be incorporated is not intrinsic to the WoC, but is a statistical question about what precisely the Crowd is saying and depends on the distribution of responses. If the Crowd is Normal, then any measure of central tendency will do. But what about when the Crowd is skewed (stock marker bubble)? or bimodal (politics)? In such cases, I don’t think it’s clear where to look (statistically speaking) for the Wisdom of the Crowd.

    Reply

  38. Camille Says:

    I enjoy your skepticism of Lehrer – read ‘How We Decide’ and didn’t have to work to notice the sheer oversimplification of the concepts he was writing about. Thanks for your frankness on non-scientists, befuddlement, accepting limits, and not being a neuroscientist. One nitpicky thing most people will probably disregard:

    Perhaps the scientist in me – who well knows how long it takes to produce these damn papers – wanted to give that faceless man in Switzerland his due.

    Could’ve been a woman too.

    Reply

    • Peter Freed, M.D. Says:

      Good point! I was fooled by the fact that I knew Jan Lorenz is a man.

      Reply

    • Voja Says:

      Very good advice. Prior to rinedag this entry, I thought learning exclusively from making our own mistakes rather than from advice or instruction was solely a Malik family trait. You mean it applies to others as well?For what it’s worth, I agree that it takes a year in any job to fully master it one complete cycle. And then after three years, time to move on.

      Reply

    • Molly Says:

      When Quotes Chimp fill out an insurance application, you make repre�sentations to the company concerning facts about yourself. Many of these facts are vital to the underwriting process�for example, whether you smoke, in the case of a life insurance policy. If you lie about any of these facts and the company catches you, and if a court concludes that the lie was material (important) to the underwriting decisions of the company, you could lose some benefits or the company may be allowed to cancel the policy.

      Reply

    • Philly Says:

      Marital status. Married drivers are statistically less likely to have accidents than single drivers�proving, if nothing else, that two can live more cheaply than QuotesChimp, at least when it comes to auto insurance pre�miums.

      Reply

  39. Max Taylor Says:

    I very much enjoyed your analysis of Jonah Lehrer’s article, all the more because it showed how you integrated a perception of your adaptive unconscious. I have enjoyed Jonah Lehrer’s books as well as Malcolm Gladwell’s (and have been much influenced by the latter, esp. Blink) and I appreciated the level of scrutiny you brought to the data. I now follow the Neuroself RSS feed.

    Reply

    • Yasmin Says:

      My coder is trying to pesaurde me to move to .net from PHP. I have always disliked the idea because of the costs. But he’s tryiong none the less. I’ve been using Movable-type on a variety of websites for about a year and am worried about switching to another platform. I have heard fantastic things about blogengine.net. Is there a way I can import all my wordpress posts into it? Any help would be really appreciated!

      Reply

  40. David F Says:

    I think your critique is off the mark in two important ways. One, Lehrer was using that statistic to illustrate a non-central point, which is why it doesn’t really matter if it was a true representative of wisdom of crowds or just another instance. He was trying to show people how the effect can work, and it often does work as has been demonstrated countless times. By neglecting to mention this in your very lengthy post, you are doing a disservice to your readers and to Lehrer, who I think is a fine journalist. Secondly, as many others have pointed out, you seem to be lacking even a rudimentary understanding of statistics.

    Reply

    • Peter Freed, M.D. Says:

      (sigh). You are in good company my friend! I have added a poll on this in my latest post. Thank you for reading! For what it’s worth, I was not dinging Lehrer as a journalist, only this one article.

      Reply

      • cathy kerr Says:

        It is troubling that non-scientists coming to your blog are cheering on your critique of Lehrer based on your credentials as a scientist when in fact (based on the “peer-review” of your commenters) what you have taught them is a marginal cranky view of statistics (eg., median is not a group statistic).

        I know and respect that you are trying to educate and give non-scientists the tools to evaluate journalistic reports on neuroscience…. I’m curious to see your reflections on this episode.

        Reply

        • Peter Freed, M.D. Says:

          Thank you! In the next post I will make the point that psychologically, medians are analogous to representative democracy: the group “elects” one (or two) representatives, far from either tail, to represent the group in public. The specific views of every other member of the group, no matter how extreme, are not thereby represented. The difference hinges on whether one thinks about the group from the inside or the outside. I’ll be interested in your feedback on the next post.

          Reply

        • Tim Says:

          Oh come off it. The point made in this blog post was that median is not representative of the group as a whole… which is basically stats 101.

          I should point out though that the entire table which you’ve shown above is useless for trying to assess the accuracy of groups because it only has measures of central tendency. Let’s be honest, both mean and median are useless on their own, particularly for this data set. I mean, look at how skewed the data is for question #6: the median is 4,000 and the mean is >135,000.

          Actually…wow… I honestly don’t even trust the data for this study. In order to get that kind of skew there would have to be a bunch of participants guessing that there were millions of assaults occurring in Switzerland in 2006… That’s more assaults than the USA had in 2006! And Switzerland has a population of less than 8 million or 3% of the US population! Either they used a very stupid sample (Swiss college students?) or they had a lot of students messing around with their responses. Honestly, I’d like to see how replicable this study is.

          Reply

          • Peter Freed, M.D. Says:

            Thanks. In what has been referred to as my “cranky” view, medians are analogous to representative democracies, in which each data point is “worth” the same amount (one data point, one vote, I suppose.) Means are weighted by the value of each vote, and thus some votes are “worth” more than others. I will get to this in a new post.

            Reply

          • www.lean-engine.com Says:

            That’s the thinking of a creative mind

            Reply

  41. ...sleeprunning... Says:

    We criticize JL. He needs to go back to school and learn a lot more and dial down the social/ideological/political polemics cloaked in pop sci journalism. David Brooks is even worse.

    However, no one will pay, money for or attention to, that kind of writing.

    There is no need to worry about these pop neuro-journalists celebrities’ image. If they didn’t exist our brains would invent them.

    Apparently the Bristish government ministers are all agog over Brooks’ silly book. All animal brains have evolved for quick, easy answers.

    Reply

    • Peter Freed, M.D. Says:

      I think I said this to another one of your posts – want to review his book?

      Reply

      • sleeprunning Says:

        Don’t know if we can stomach it. Witch one? Writers get paid by the word, or actually eyeball now, and everyone needs to earn a living. In addition if J Lehrer didn’t exist – our brains would invest him. Wait! Maybe we already have?

        But “never before in human history” (there’s some hyperbole) have we faced the urgent necessity of integrating the best science with general public discourse and policy. Mainly because science has gotten so darn good, at last cracking the brain, genomics, etc. for example, and there are so bloody many of us.

        So, we propose, what is published in Science and Nature and even specialized journals:
        – Is now available to all — instantly — and is not longer mediated by others
        – Can have immediate personal and policy implications

        But (maybe) this means the public — not pop — reader does need to be smarter about filtering and critical thinking. Like it or not basic stats understanding is needed. But a new understanding of stats one that’s not headline worthy but also not fully professional. Ugh. Sounds like work — for us all.

        The nice thing about the post is that the author’s voice takes a stand between the professional and the pop and says: “Wait a minute! What can we really say about the original research and the WSJ article? That is a very new ( and kool) set of questions. Questions that (sort of) haven’t been important before. Now they are — very important. But it’s never fun being in the middle.

        Reply

  42. Thiri Lin Says:

    Dear Sir,

    I’m not sure if someone has already addressed this, I didn’t read all the 80 something comments. But I wanted to share my point of view. I AM a Jonah Lehrer’s fan, one of the “consumers of pop neuroscience”, and I’m not a scientist (yet). English is not my first language and I’ve only been here for 6 years, so be ware, I can’t really brilliantly write out my point like you or Jonah Lehrer.

    I’m offended that you and some of the scientists really look down on people who read popular science and usually very unimpressed by popular science. I am passionate about brain science and I am going to grad school to learn more about it this fall. And I have to say, Science journalists such as Jonah Lehrer, Carl Zimmer… Podcasts like Radiolab are one of the big factors that inspired me to become a neuroscientist and I’m not ashamed to admit it. Not all pop science/neuroscience fans are stupid and try to makes sense of the brain easily. I AM AWARE of the power and complexity of the brain and KNOW that we have a long way to go to even understand 10% of it. Of course, there is good pop science and bad.. really really terrible pop science, so I’m not saying your are completely wrong. Jonah Lehrer is one of my favorite writers, and yess.. like a very typical consumer of pop neuroscience, I picked up his book in Borders a few years back because of the witty title. When I read popular science I’m not believing all the flawless data and thinking that we can explain human behaviors and brain as easy as “lying in bed and remembering that madeleine”. I know very little and good popular science is a great start. I did learn a lot from Jonah Lehrer’s writing’s, such as some interesting history of science, names of great neuroscientist that I’ve never heard of and their studies and thought-provoking scientific questions. I would go back and research the paper or the facts he talked about and which led me to look into many other interesting things .. which makes me want to explore more and makes me want to be a scientist. People can take in all kinds of different things from reading popular science; not ALL the fans of Jonah Lehrer and consumers of pop neuroscience are stupid like some of the “real scientists” think. For example, the only thing I took in from reading your blog article is the part where you mentioned how you read primary articles, “skip the intro, go straight to the tables and figures, and then to the methods.  If you ever read a science paper, you should do the same thing yourself. Reading intros and conclusions first is for suckers”.. you are a scientist, so I thought this is a great advise and I might try it next time. So I learned something from you but I didn’t take in any of your critique of Jonah Lehrer. Oh and how dare you compare Jonah Lehrer’s writing with snooki!!! (ew).

    And I think that Science needs imagination and people like Jonah Lehrer and Podcasts like Radiolab. I like how these science journalists think and speculate about complex scientific questions. Of course not all of them, but some of the science journalists/writers are brilliant thinkers. I couldn’t put it any better than Dr. V.S. Ramachandran from the preface of his recent book The Tell-Tale Brain:

    “Speaking of accuracy, let me be the first to point out that some of the ideas I present in this book are shall we say, on the speculative side. Many of the chapters rest on solid foundations, such as my work on phantom limbs, visual perception, synesthesia, and the Capgras delusion. But I also tackle a few elusive and less well-charted topics, such as the origins of art and the nature of self-awareness. In such cases I have let educated guess work and intuition steer my thinking wherever solid empirical data are spotty. This is nothing to be ashamed of: Every virgin area of scientific inquiry must first be explored in this way. It is a fundamental element of the scientific process that when data are scarce or sketchy and existing theories are anemic, scientists must brainstorm. We need to roll out our best hypotheses, hunches, and hare-brained, half-baked intuitions, and then rack out brains for ways to test them. You see this all the time in history of science. For instance, one of the earliest models of the atom likened it to plum pudding, with electrons nested like plums in the thick “batter” of the atom. A few decades later physicists were thinking of atoms as miniature solar systems, with orderly electrons that orbit the nucleus like planets around a star. Each of these models was useful, and each got us a little bit closer to the final (or at least, the current) truth. So it goes. In my own field my colleagues and I are making our best effort to advance our understanding of some truly mysterious and hard-to-pin-down faculties. As the biologist Peter Medawar pointed out, “All good science emerges from an imaginative conceptions of what might be true.” I realize, however, that in spite of this disclaimer I will probably annoy at least some of my colleagues. But as Lord Reith, the first director-general of the BBC, once pointed out, “There are some people whom it is one’s duty to annoy.””

    So yes, even if I become a scientist one day, I would never look down on non-scientists readers of popular science and science journalists. And I will still be an imaginative scientist and open to all kinds of thinking process. Sorry it got too long. But thank you for reading it (if you did). You can correct me if some of my thinking are wrong. After all, this is just my opinion that my 23 y.o. brain came up with. There are many things I do not know and I’ve got a lot to learn. And can I save the cover-image of the pigeons and use it? It’s a great photo!

    Sincerely,
    Thiri

    Reply

  43. medskep Says:

    After going through the comments and all the statistical discussion I want to circle back to the important use of the phrase “for instance.”

    When ‘for instance’ is used to describe one piece of data within a data set, most (probably all) readers will be led to the unspoken assumption that the example is representative of most, if not all, of the members of the data set.,

    This use of language doesn’t have much to do with sophisticate statistical concepts but is vitally important when analyzing the rhetoric.

    btw, I like to read Jonah Lehrer and feel that criticism such as yours will help make him an even more important witness of neuroscience developments.

    Reply

    • Peter Freed, M.D. Says:

      Thank you! That was my main point, that the WSJ readers were being inadvertently misled about what the overall stats showed. When Galton looked at the mean of the guesses of the ox’s weight, it was off by 1 pound in over a thousand. I think Lehrer was probably just trying to find an “example” like that. Intriguingly, Galton used the mean (per Wikipedia – I don’t know this independently) – but in this study the means were way, way off.

      Reply

    • Peter Freed, M.D. Says:

      Right! I want to start very carefully vetting pop science – want to help?

      Reply

  44. sleeprunning Says:

    Lookee here, lookee here: why-dont-journalists-link-to-primary-sources/

    http://www.badscience.net/2011/03/why-dont-journalists-link-to-primary-sources/

    Reply

  45. Tom Says:

    The median is not a group statistic? Oh my. Anyone who is a bona fide scientist must have a much better grasp of statistics than what is conveyed in this blog article. It takes you minutes to comprehend the full meaning of a statistical table like this? You didn’t realize the implications here of the elementary concept “median” within, like, 3 seconds or so of looking at the table? Surely you jest.

    Given the ubiquity of unsatisfactory statistics competence in the brain sciences, I really wish they would institute a compulsory “statistics license” for everyone who wish to publish, preferrably one that is subject to testing and renewal every other year or so.

    Reply

    • Pearl Says:

      Please let me know if you’re looking for a witrer for your weblog. You have some really good posts and I believe I would be a good asset. If you ever want to take some of the load off, I’d love to write some content for your blog in exchange for a link back to mine. Please send me an e-mail if interested. Thank you!

      Reply

  46. gonzalopolavieja Says:

    If you go to Galton´s original paper ‘Vox populi’ , you will see he favors the median as the best way of aggregating the crowd´s answers. He does so because the median does not weight more those giving answers far too distant from the true answer, as means do in skewed distributions. Also, note that the median, even if it corresponds to what one person says, is defined by the group´s answers, so it is a true statistical number.

    Reply

  47. Blackjack Says:

    Shocked after reading the post, never got to know about the facts of switzerland. I was looking to do my masters in zurich actually.

    Reply

  48. resiliencia Says:

    I’ve understand quite a few excellent products in this article. Definitely worth book-marking intended for returning to. We shock the best way a great deal attempt you placed to generate the fantastic useful web site.

    Reply

  49. how to get bigger breasts fast Says:

    Does your site have a contact page? I’m having a tough time locating it but, I’d like to send you an email.
    I’ve got some ideas for your blog you might be interested in hearing.
    Either way, great site and I look forward to seeing it grow over time.

    Reply

Trackbacks/Pingbacks

  1. Online Poll: On the Merits of Screwing Up | Neuroself - June 3, 2011

    [...] on some of the comments I have received, my post about Jonah Lehrer’s WSJ article seems to have been my first screw-up moment with the [...]

  2. The Neuro Week 5th June 2011 | Health Book - June 5, 2011

    [...] “Proust Was A Neuroscientist“ Lehrer is not a neuroscientist… and he’s quoted some dodgy data on the ‘wisdom of crowds’, says new blog [...]

  3. Open Laboratory 2011 – submissions so far | A Blog Around The Clock - June 6, 2011

    [...] Neuroself: Jonah Lehrer is not a neuroscientist [...]

  4. Chasing Their Tails: When Central Tendency Junkies Attack, Part 1 | Neuroself - June 7, 2011

    [...] about Zach until last week, when I posted an article on this site with the milquetoast title “Jonah Lehrer is Not a Neuroscientist.”  In it I sought to illustrate the perils of pop neuroscience, and the risk that when boiling down [...]

  5. ¿Es viable el “crowdstorming”? | La Pastilla Roja - June 30, 2011

    [...] un neurocientífico de Columbia publicó un post poniéndole a caer de un guindo y afirmando que las estimaciones de la muchedumbre eran en realidad lo peor de lo peor. La trampa, en mi opinión, es que todos los experimentos de estimación estaban seriamente [...]

  6. 3×5: Culture, Neuroscience, and Psychiatry Roundup | thefpr.org blog - June 5, 2012

    [...]  Proust wasn’t a neuroscientist, and neither is Jonah Leher. An old, but very rich post + commentary, which may make you want to revisit the discussion between [...]

  7. Attack of the killer stats assignment | Bad Blood - September 20, 2013

    […] Attack, which was inspired by the vitriolic response to this, now rather prescient-seeming post, Jonah Lehrer Is Not A Neuroscientist.  TL;DR: Freed argues that statistics should be understood as metaphysics, not a science.  This […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 542 other followers

%d bloggers like this: