You could argue that they are making conclusions about all apples from a sample of golden delicious apples, and I would agree with you. But to claim that this is apples and oranges is a bit much.

Ok, you can reduce the scale of my analogy if you like, but my point is still valid. The subset is non-random* and therefore it is not logically possible to draw conclusions about the whole from that subset; if you want to make a hypothesis from the subset and use that as the basis for further experiments on the whole, knock yourself out. You can’t, however, make conclusions.

*Not only is it not random, it isn’t even mixed up a little. The subset is narrowly defined to only include a single class of cases.

Now, again, you could argue that the statistics they present, on their own do not back up this claim.

That’s what I was doing.

With that being said, they are absolutely correct that eyewitness testimony can be quite unreliable

If you actually follow the link, you will see that TIP considers such factors as estimator variables, and system variables. TIP also has made observations about eyewitness misidentification based on the over 100 convictions they have helped overturn.

I don’t necessarily take issue with their conclusion. I just take issue with the way that stat was presented. It struck me as MSM-style sensationalism.

Even if the stat did speak directly toward their conclusion, they still should have given it some context. “Played a role in more than 75%”? Ok, where does that fall in comparison to other factors? Presumably there are multiple factors in most wrongful convictions, so 75% might not even be good enough for 3rd place for all I know.

This case, and the issue of eyewitness testimony, reminds me of two classes I took, one in college and one in law school, where the professor engaged in an experiment that showed just how unreliable eyewitness testimony can be.

It should be familiar to anyone who’s taken a criminal justice class of any kind.

In both cases, we witnessed what looked like a crime (theft in one case, attempted assault in the other) but was actually a ruse between the professor and a graduate assistant. After it happened, the professor asked the class to recount details about what they saw such as:

- The height of the person who had run into the room
- What color hair they had.
- How tall they were
- Whether they were fat or thin
- What was said

Stuff like that.

Without exception, everyone got at least one crucial detail about the incident completely wrong. And that was mere minutes after it happened.

I don’t practice a lot of criminal law, but even on the civil side it became obvious to me quite early on that when someone gets up on the stand in a courtroom and is asked to recount the details of an event that happened years in the past, they usually aren’t getting everything completely right.

In a civil case, it usually isn’t crucial because documents usually control.

In a criminal case, where someone’s freedom or life is on the line, basing a conviction solely on the testimony of witnesses, without corroborating physical evidence seems foolish.

If you actually follow the link, you will see that TIP considers such factors as estimator variables, and system variables. TIP also has made observations about eyewitness misidentification based on the over 100 convictions they have helped overturn. If you look to the right side of the web page, you will find more outside information about the unreliability of the use of eyewitness identification: here’s an Iowa University Study and a website from one of the study’s authors Iowa State Psychology Professor Gary Wells

If you are skeptical about TIP’s sources, do an independent Google Scholar Search on the topic from 2000 to 2008. You will find that Dr. Wells’ articles are among the top scholarly articles cited by other peer reviewed scholarly articles found by Google Scholar.

On Table 1 of this study which interviewed 64 psychologists who had courtroom experience found in the May 2001 issue of American Psychologist, the very first of thirty variables listed in determining the reliability of an eyewitness is the stress level at the time of the event. The statement next to this variable reads as follows: “Very high levels of stress impair the accuracy of eyewitness testimony.”

This would explain why I could not remember exactly what this man looked like even though he was looking right at me less than 2 feet away; I was scared shitless (to use a scientific term).

The very first page of the same study states the following:

In 1996, the National Institute of Justice reported on 28 wrongful convictions, cases in which convicted felons were exonerated by DNA evidence […] Remarkably, all of these cases contained one or more false identifications.

This is only the tip of the iceberg. The body of research being done concerning eyewitness mistakes is growing. Read these studies as well as a few of your own and then tell me that eyewitness testimony is generally reliable enough to take someone’s freedom or someone’s life.

These aren’t apples and oranges. The convictions overturned is a subset of a larger, unknown and unknowable set, all the wrongful convictions. The use of a small sample to come to conclusions concerning larger sets is the heart of “hypothesis testing” which is one of the most highly developed branches of statistics.

You could argue that they are making conclusions about all apples from a sample of golden delicious apples, and I would agree with you. But to claim that this is apples and oranges is a bit much.

With that being said, they are absolutely correct that eyewitness testimony can be quite unreliable. In fact some branches of eyewitness testimony are known to be often unreliable, such as identification of a stranger. There was a series of psychological experiments that demonstrated this; the researchers could consistently get witnesses to finger the wrong man as a person who committed a a simulated robbery merely through having a patsy in the crowd loudly and incorrectly describe one aspect of the thief’s appearance.

Now, again, you could argue that the statistics they present, on their own do not back up this claim. However, the unreliability of witnesses when describing novel people or situations is well known enough that it needs no confirmation anymore than a guy talking about airplane design need to justify his assertion that air pressure can be used to calculate altitude.

If you read a report analyzing aircraft crashes, and the report stated that cracks in the pressure hull had contributed to 75% of crashes, I doubt you would be claiming that that statistic was somehow misleading and that the important question was how many planes flew with similar cracks safely.

It depends on the point you’re trying to make.
Read that sentence again:

Eyewitness misidentification is the single greatest cause of wrongful convictions nationwide, playing a role in more than 75% of convictions overturned through DNA testing.

Notice that the assertion is about “wrongful convictions” (herein referred to as “apple”), yet their supporting statistic is about “convictions overturned” (herein referred to as “orange”).

We can agree that there are some unknown number of wrongful convictions that never get overturned, right? Isn’t it quite possible that there are other unmentioned factors that bias the sample?

And even if there aren’t, the author goes on to assert that eyewitness testimony is “often unreliable.” He didn’t offer a shred of evidence to support that statement. The eyewitness testimony could be right 99.999% of the time for all we know, because he only examined the convictions that were overturned.

In the context that it was presented, the stat is useless at best.

If you look at a failure of a system, looking at the sources of error and their frequencies is quite useful.

If you read a report analyzing aircraft crashes, and the report stated that cracks in the pressure hull had contributed to 75% of crashes, I doubt you would be claiming that that statistic was somehow misleading and that the important question was how many planes flew with similar cracks safely.

The proper denominator would be the number of convictions that are subsequently tested where eyewitness identification played a role.

I would also include those convicts for whom DNA evidence was available, but never tested. If a convict isn’t pursuing all possible means for an acquital, it’s hard to assume he’s anything other than guilty.

Regardless, I think we can agree that the statistic cited is pure garbarge.

The relevant stat would be the percentage of convictions based on eyewitness identification that were later overturned due to DNA testing.

I don’t completely agree. A very small portion of convictions are amenable to DNA testing. The proper denominator would be the number of convictions that are subsequently tested where eyewitness identification played a role.

The problem with examining the cases tested by the innocence project is that there will be a sampling bias: people who are innocent are much more motivates to seek out avenues for exoneration, and the innocence project will select cases that where they have a higher probability of success to focus their limited resources.

Defense lawyers and investigators badgering the witnesses to recant, guilt trips levied on witnesses by them, and in at least one case (Culhane) false statements.

Let us remind these recanters that their original statements were perjury and “should be prosecuted” (yes, I know about the statute of limitations).

Eyewitness misidentification is the single greatest cause of wrongful convictions nationwide, playing a role in more than 75% of convictions overturned through DNA testing.

That’s a misleading stat. The relevant stat would be the percentage of convictions based on eyewitness identification that were later overturned due to DNA testing.