Caveat: Implicit Association Tests

I found a website (named "Project Implicit," by something called IAT Corp, hosted at Harvard) that makes some claim to evaluate the kind of unconscious mental associations between categories like race, gender, sexual orientation, etc., and other semantic fields (like good vs. bad, American vs. not-American, etc.).

You do these rapid response categorization tests and then the test tells you how you tend to lean in your alleged "automatic preferences."  I harbor all kinds of skepticism about this sort of test, on multiple counts.  I might discuss some of these skepticisms later, but for now, I'll present my personal results on two of the tests (in the spirit of disclosure and for those curious).

The first test I took was with regard to the African-American category (Black) vis-a-vis the European-American category (White).  Impressionistically, the alternation between labelling as Black vs. African-American on the one hand and White vs. European-American on the other hand struck me as inconsistent or random, although I can't say for sure that wasn't a designed inconsistency (e.g. something intentionally random as a built-in part of the test's brain-probe, so to speak).

Below is the interpretation of your IAT performance, followed by questions about what you think it means. The next page explains the task and has more information such as a summary of what most people show on this IAT.

Your Result Your data suggest a slight automatic preference for African American compared to European American.

The interpretation is described as 'automatic preference for European American' if you responded faster when European American faces and Good words were classified with the same key than when African American faces and Good words were classified with the same key. Depending on the magnitude of your result, your automatic preference may be described as 'slight', 'moderate', 'strong', or 'little to no preference'. Alternatively, you may have received feedback that 'there were too many errors to determine a result'.

I quickly felt that I was aware of "how" the test worked - it's hard to explain so I suggest you just try it for yourself.  I admit that from the start, I felt wary (on gaurd, so to speak) with regard to my own possible prejudice, and once I felt I understood how the test worked, I perhaps attempted to compensate.  Assuming that the underlying prejudice I presumed myself to be battling (as a White American raised in a 90%+ white community) was one of preference toward European-Americans, it appears (and I can only say "appears" as I hardly know what all was operating, both in the test and in my own brain) I compensated successfully.

I found the first test unpleasant.  The business of matching Whites with "Good" words and Blacks with "Bad" words (and then subsequently vice-versa) left a bad taste in my mouth.  It was like the underlying message was:  "everyone's a racist, we just want to see what kind you are."  It was an exercise in reinforcing stereotypes, whether positive ones or bad ones.

The second test wasn't really unpleasant so much as downright ridiculous.  It was supposed to look at the European-American/Native-American contrast vis-a-vis the American/un-American (Foreign) contrast.  The visual images drew on stereotypes even worse than the first test (see screenshot below).  Of course, stereotypes are the point, and therefore it's utterly conceivable that they're intentional.  Still, it's awkward for someone who tries to be analytical about these things.

The whole business of what words were "American" vs. "Foreign" struck me as silly - they were all place names - essentially, European place names versus American place-names of Native American etymology. What is this contrast supposed to show?  That Americans know the names of American cities?  What about the allegedly atrocious geographical knowledge of average Americans?  Is this test trying to link bad geographical knowledge with some type of racial (or racist) stereotype or another?  Or is it assuming good geographical knowledge?  They're aware that Miami is in Latin America, right?  And that Seattle is in Canada?  And Moscow is "Foreign" - but what about the guy sitting in Moscow, Idaho, taking the test?  I've been there.  It's near the Nez Perce Reservation.  Did they take that into account?

What does this test really mean?  What is it looking at?  What does it have to do with nativism, white-supremicism, pro- vs. anti-immigration stances, etc.?  It's obviously complex, but I felt immediately that the test designers had at least as much ideological baggage as I personally brought to the table, and they didn't even do much work to conceal it.  I certainly doubt they had made much effort to evaluate their own prejudices, in the design of the test (especially in light of the apparent socio-linguistic naivety on display in the onomastics).

I felt a strong impulse to try my best to "game" the test.  I have no idea whether my effort to game the test worked, but it appears to have, since I got the result I intended: I got myself to show up as a nativist, roughly.  But of course, the test designers could argue that I was merely "aiming for" the "automatic preference" I was already ideologically inclined toward.  Here is my result.

Below is the interpretation of your IAT performance, followed by questions about what you think it means. The next page explains the task and has more information such as a summary of what most people show on this IAT.

Your Result Your data suggest a moderate association of White Am. with Foreign and Native Am. with American compared to Native Am. with Foreign and White Am. with American.

The interpretation is described as 'automatic association between White Am. and American' if you responded faster when White Am. images and American were classified with the same key than when White Am. images and Foreign were classified with the same key. Depending on the magnitude of your result, your automatic association may be described as 'slight', 'moderate', 'strong', or 'little to no preference'. Alternatively, you may have received feedback that 'there were too many errors to determine a result'.

So what does it all mean?  I'm not sure.  I might take some more tests and report back - they're nothing if not interesting.

 

Comments

[Brenda]

Lots of research using brain scans have gone into this stuff, Jared. I've been following the research, and I guess contributing to it periodically, as well. It's pretty well-founded, as far as social science research goes.

[Jared]

Hi Brenda.

I really appreciate your comment. And... I in no way dispute that a great deal of research goes into these - that's obvious. But I still believe there's a lot going on, ideologically, that isn't being taken into account by the tests' scoring systems and/or by the creators.

After seeing your comment, out of curiosity, I re-took the Native-American/European-American axis test. Because of my intuition that I was able to "game" the test - meaning that I could force a result in a consciously-chosen direction. In retaking the test, I said ahead of time: "I want to produce an "opposite" result of my last time. I took the test, and my result was:
"Your data suggest a moderate association of Native Am. with Foreign and White Am. with American compared to White Am. with Foreign and Native Am. with American."

I think this confirms my instinct that the test is "cheatable" - which in turn calls into question its effectiveness.

I didn't "game" the test by thinking particularly consciously about how the test worked. I simply focused on the result I wanted, and allowed my reflexes to produce them. Obviously, a great deal of work would be required to determine if my result is reproducible - by me or by a broader population.

Actually what interests me (and concerns me) more isn't how "reliable" the results are but rather the many errors I perceive at what might be called the socio-linguistic level. Maybe, not so much errors as oversimplifications of the way language and culture and prejudice interact. Who are the linguists involved in this research? One of the huge weaknesses in so many of the social sciences and psychological research are the way that the researches either inadvertently overlook or willfully ignore the many findings of psycholinguistics, sociolinguistics, linguistic pragmatics and even fairly abstract areas like field semantics.

Again, please understand, this blog post wasn't, in fact, meant as criticism - I'm hardly an expert. I was merely identifying some aspects of the tests that made me uncomfortable intellectually.

Regardless, I appreciate your thoughtful reading. And by the way, more personally - I have always appreciated the kindness/friendship you showed me, those years ago, during a difficult time. Take care ^_^