Not only WEIRD, but DEAD
New psychological science acronym just dropped.
As developed the Crockett lab’s recent paper “AI Surrogates and illusions of generalizability in cognitive science”.
You might already be aware of the WEIRD sample critique - that psych science has tended to experiment on “convience samples” of people that are unrepresentative of the world’s population. That is to say folk that are:
- Western
- Educated
- Industrialized,
- Rich
- Democratic
Doing so limits generalisability. How could you possibly say that your finding applies to humans in general if you’ve only tested it on Harvard undergrads? But yet, few papers present their findings as “we found X in Harvard undergrads”, preferring to make sweeping statements about humanity.
Crockett and Messeri have come up with another you-can’t-generalise based acronym. DEAD:
- Decontextualized: interpreted without taking into account any cultural, relational, institutional, historical or situational factors participants are affected by.
- Engineered: measured using tasks that are designed to suit the limitations of what the researchers can easily measure rather than the real experience of someone in the outside world.
- Anonymized: taking no account of who is being measured; their identities, life experience, social positions or other relationships.
- Disembodied: without reference to any effect that people’s bodies, physical surroundings and so on might play on cognitition.
This is again a generalisation based critique, but instead of WEIRD’s focus on who gets studied, it relates to exactly how things get studied.
Work in the cognitive psychology field is full of proxies designed to measure some more general - and more interesting - psychological construct. The authors' critique here is that with DEAD data you cannot be at all sure that the actual thing you measured properly reflects the in-context human experience of the construct you want to write about.
In their words:
DEAD cognition is unlikely to be representative of the diverse ways in which cognition lives out in the real world: contextualized in culture and relationships, sprawling and unconstrained, tethered to our identities, and enacted through our bodies.
An example is instructive. There are a lot of papers that claim to measure (or alter) how generous humans are. But “generous”, in the layman’s sense of the word and its complete meaning, is obviously a rather difficult thing to measure quickly and simply in a lab or online. So instead, proxies such as the Dictator Game are used.
In this:
… participants are invited to share money with a recipient (typically a stranger or charity). Giving money is taken as a measure of generosity…
But:
…this task is hardly representative of all the ways in which humans show generosity. The standard version of this task measures generosity outside of specific relationships (decontextualized); it allows only one way to express generosity (engineered); articipants are typically not identified to experimenters or recipients (anonymized); and it excludes embodied cues and expressions (disembodied).
However, the resuilting papers tend to want to claim that “our intervention makes people more generous”, not “our intervention makes people behave a bit differently in a simple game involving a low sum of money”.
This is thus a potentially huge generalisation issue:
When we write papers about abstracted processes, such as ‘generosity’ (rather than, say, ‘donating US$1 to the Red Cross over the internet’), we perpetuate another illusion of generalizability, portraying DEAD cognition as more representative of cognition than it actually is.
A useful discussion in this episode of Two Psychologists Four Beers elaborates further. To provide an exaggerated-by-me version of one of their examples: Some unknown Mechanical Turker who decides to keep the $1 rather than donate it could be reflecting real selfishness of a millionare who craves every last bit of money, but on the other hand they might be living out of their car, desperately saving money for their mother’s life saving operations. Laypeople would likely interpret those two cases as rather different representations of generoisty, but the typical psychological study does not even attempt to differentiate between the two.