In the large telephone survey of 65-year-olds that I work on, there is a task in which respondents are asked to name as many animals as they can in one minute. Tonight I'm working on the computer code that processes responses that have been coded as erroneous. Here are ten examples of "animals" named by respondents:
amarillo
bamboo
gazebo (3 different people said this!)
jalapeno
lummox
maggie
milk
onyx
potatoes
scarecrow
Subscribe to:
Post Comments (Atom)
20 comments:
A gazebo is a gay zebra, everyone knows that! HA HA HA
You get some real wags here. How about a quick synopsis of what this survey is all about?
Aw. You can see what they meant: magpie, baboon, gazelle, oryx, armadillo, mink. How can you be sure the person taking down the responses wasn't mishearing or mistyping?
you "work on" 65-year-olds?
Skewing the geezers:
For cryin' out loud! anyone knows what when a Senior has his or her teeth out and they say what you think is onyx is really 'old ox'. Likewise lummox is a 'lame ox'. Some older Polish immigrants refer to a cow as ' a milk'. Some Hispaics coming out of the Guadalajara area refer to dogs as 'jalepenos'. Factor that, buster.
I've wondered about, for example, the 'onyx' and 'oryx' issue, eespecially because the coders may or may not know what an oryx is. Some coders coded "fisher" as an error, despite its being an animal. Anyway, all of the interviews are digitally recorded and transcribed from the recordings, so, in principle, one could go back and check any of these for a possible mishearing. But, errors are sufficiently rare that my guess is that a coder listens to it a few times ("Dude, did that person really say 'bamboo'?)
Goesh: The survey is a general survey that asks about all kinds of different things. The name as many ____ in one minute part of the survey is supposed to provide a measure of verbal fluency.
Two questions that would help resolve Ann's issue. Is there any auditing of the interviews, and who's coding the responses as erroneous? "Potatoes" may need to se a neurologist, but "amarillo" could plausibly be either data collector error or simple malapropism.
As for the underlying issue of data collector reliability, it's real, I can tell you. My First Law of Surveys (ordinal position subject to change when I bother codifying the rest of the laws) is that for any sufficiently large survey, data collector error will at some point drive the most mild-mannered researcher at least to fleeting moments of homicidal insanity.
Wow, my questions were answered with causality-breaking speed. Thanks!
Yeah Jeremy, I agree with the other commenters that raggin' on the participants without trying to interpret their answers is not cool.
But the commenters also need to understand that the urge to laugh is sometimes irresistable--participants say the darnedest things, which is particularly hilarious to the researcher who has just spent hours (often hours when other folk are sleeping) looking at data that are interesting but not exactly Comedy Central. If Jeremy couldn't laugh at people who identify "potato" as an animal, he might crack from the stress and be found wandering around State Street muttering about gazebos. I suspect that Jeremy is busy trying to decipher these answers exactly along the lines you suggested.
I ran some questionnaires last semester, all open-ended questions. One woman (age 79) wrote that the thing she appreciates most about her best friend is that, "She keeps me mentally challenged."
Ok, everybody understands what the participant meant, and in the same situation I might have written the same thing, especially if I belonged to a cohort in which "mentally challenged" is not a politically correct way of saying "mentally retarded."
But...HEE! With friends like that, who needs enemies?
--Evil Glitter Princess
I suppose we can assume that the *geezers* are too busy using computers for gelealogy searches to bother tracking blogs of younger people, so it's not as if anyone in the affected category is going to be swinging by JFW and feeling hurt reading this.
My question is this (in light of Tom's comment about a neurologist for anyone who says potatoes): if there is no penalty for saying something stupid, wouldn't anyone at any age just blurt out endless animal-like (and freely associated agricultural, farm, nature, etc) words and not hold back? I wanted to see how many animals I could list in a minute (I was taking a pause from a genealogy search to exercise my mind, which, in 14 years will qualify as a candidate for the survey)and I have to admit, I was thinking jackal but said jackass, which in the end, IS an animal, but not one I wanted to hurl out there.
--nlc
I do want to be plain, especially as someone who produces my own share of malaprops both in speaking and typing, that I do not intend AT ALL to be "ridiculing" survey respondents by noting that a few of them, occasionally, say things that are kinda funny. Given the amount of aggregate time that respondents have put into doing our survey, we owe them considerable respect, and I have considerable respect for them.
Such responses, though, have required considerable effort on the part of people who work with survey data about what to do with them. In general, my work was trying to come up with scores for this fluency task has taken A LOT of cognitive labor, and most of that labor has been taken up with dealing with different kinds of responses that are, in total, maybe less than 5% of all responses provided. It's just that, when you are surveying several thousand people, even infrequent things add up fast.
We have an excellent student over here who works with me on the verbal fluency data. She just re-listened to several of the cases listed below, and, indeed, in all instances, the person did say what we have them down as saying.
In some cases, clearly, the person had a different word in mind. In the case of "bamboo", the person who said it even realized it was wrong and said something like "I'm trying to think of an ape." Of course, if your interest is the cognition of language production, there is a giant difference between being able to produce ape-with-a-name-like-bamboo and "baboon". Especially as people start getting older, the tip-of-the-tongue memory problem becomes more profound, where people have a general idea of a word but can't think of the specific word itself.
Ultimately, there some tricky issues here about psychological measurement that turn on exactly what it is we are hoping to measure by measuring verbal fluency. In terms of speed of production, it's not at all clear to me that there is any reason to make a distinction between somebody who says "amarillo" and those who say "armadillo." For other conceptions of verbal fluency, there is. We'll be providing all kinds of information for data users to construct their own scores, so can make their own decisions here in some respects.
By God any voice over the phone that sounds like a scared crow or a lame ox or a nervous armadillo and pesters geezers with said timbre and inflection is in turn going to elicit the odd, occasional response. Stir that in your Geritol, buster.
I did not say "a neurologist for anyone who says potatoes." I may have been thinking it (at least with respect to 65-year-olds asked to name animals; I won't be seeking a referral for telling my son last night that there were potatoes on his dinner plate), but that's another matter.
The recordings presumably would shed some light on the free-association theory.
The tip-of-the-tongue memory problems alluded to gives me pause to consider the multitude of imagary, memory and life experiences that could and probably would accompany the simple naming of a certain animal. In the framed reference of the question, perhaps the significance of the exercise, to say a 25 yr. old ,is only in her/his ability to give a snap reply. Maybe to have lived (survived) 65 years simply mandates the constant re-laying of foundations, the imagary and memories, and doesn't necessarily preclude the ability to give snap replies. I don't know, but there is merit to what Tom Bozzo suggests and my hunch is that the recordings would show a fair amount of free association occuring.
Jeremy: I don't think that the post is at all disrespectful of the elderly, though I can certainly appreciate heightened sensitivities when thin people make remarks about Slender Fare and young people refer to older people as *geezers.* But you did not, I presume, use that term in the survey.
But you did not answer my Q, which was: what's the penalty of saying something wrong? If you are thinking that you are being judged for your numbers rather than accuracy then you will say a lot of misplaced things, including potato and gazebo. And so we may learn very little about age and verbal exactitude or fluency, depending on how the respondent was instructed, right? [Though maybe you are comparing these to a younger set and the instructions remained the same over time.]
--nlc
Nina: In the kind of verbal fluency task we are using, persons are asked to say as many words that fit a given category within a minute. The ultimate score is just the count of non-disqualified words. So, the only penalty of saying something coded as an "error" is that it won't count for your score. We do keep track of number of errors as a variable, and it's possible that could be interesting for analysis. We also keep track of number of repetitions, and it is more likely, actually, that number of repetitions might be an interesting variable, as repetitions can signal problems of either working memory or executive control over production.
I am in agreement with your point about repetitions proving to be ultimately more serviceable than incorrect insertions (for any number of reasons, some discussed above)and I did think that neither rep's or err's were the main focus of the initial survey, which is why I posed the question. And I appreciate your answer even if it was given to someone called “Nina.”
--nlc
Jeremy, here is a brief offline exchange I just had:
>Your point is a good one. The survey's script ideally ought to clarify
>the researchers' intent, since otherwise the respondent's intent is
>nonobservable.
>
>Tom
>
>
>-----Original Message-----
>From: [nlc]
>Sent: Wednesday, February 09, 2005 12:29 PM
>To: Tom Bozzo
>Subject: [some subject]
>
>Yes, but my point (in the comments) had more to do with understanding
>what respondents thought was wanted of them. Most would strive to
>perform well.
>Were they striving for accuracy or for sheer numbers? If the latter,
>then the "wrong" answers become less interesting.
>
>[nlc]
Sorry, NLC. Whenever I see your initials I always think of Nina, and I forget that I have at least two regular readers with those initials. I'm surprised Nina hasn't weighed in with her own complaint about my asserting that a commenter is her when it isn't.
Maybe she's terrified of your Comments function and publishes under a pseudonym. Though you should conclude that I can’t be her since I happen to know that she teaches a class at this hour and we already have heard that you cannot tamper with comment times.
--nlc
Post a Comment