Beware the citizen scientist!

21 08 2015

This week, an editorial in Nature addressed the now well established use of citizen science generated data. Though the line “Some professional scientists are sniffy about the role of amateurs,” the piece points out the general acceptance of the value of volunteer-driven science. But even though scientists are using or at least accepting these data (some grudgingly, perhaps, or even sniffily), the editor points out some potential pitfalls to be considered. A major one is the potential for bias introduced by a self-selected team of citizen scientists. In a program like SEANET, we know our volunteers tend to be conservation minded, outdoorsy types. Nothing wrong with that, but as the editorial describes, an interest in conserving the subject of a given study could change the behaviors of the data collectors. People’s personal predilections and preferences can also skew data. This could be deliberate, or, perhaps more likely and more difficult to detect, subconscious. In projects like eBird, researchers learned early on of the tendency of many birders to overreport rare birds and underreport common ones. Overreporting the rare birds does not mean the birders record birds they don’t actually see, just that they may report it every time they see a rare bird, and not bother to report a bird they see everyday even if it was sitting side by side with the rarity. eBird now asks the question,
“are you reporting all birds you saw/heard?” to make sure you aren’t cherry-picking, or, if you are, that they can account for it in their analyses.

img_3208For Seanetters, that particular type of skew might not seem relevant since Seanetters go out for a beach walk and record every dead bird they see, regardless of species. We don’t accept incidental reports of oddities like an albatross in the backyard. But we still wrestle with these issues. Many of our volunteers join up thinking that finding the most dead birds is the goal, and that a “good” beach is one that turns up many carcasses. Some volunteers ask if they should modify their walk schedules to coincide with the aftermath of a big storm in the interest of finding wind and wave-tossed bird bodies. That’s why we try very hard to instill in all our volunteers the value of a true baseline (or as close as we can get). For our purposes, a beach that never turns up any birds is no more or less valuable than a beach stacked high with carcasses. For the volunteers who rarely or never find dead birds though, the chance to finally  record one can be tempting. Seanetters will email from time to time saying they’ve been tipped off to the presence of a dead bird on the beach on a day when they had no walk scheduled, asking if they should go out and record it as a walk. I understand the temptation, but sticking to the set schedule of monthly, twice-monthly, or weekly walks is critical to our attempt to get at that baseline.

lc_feb2012_3The editorial closes with a mention of “increased scrutiny […] on the reliability of the work of professional scientists,” a reference, I can only assume, to recent stories of trained scientists fabricating data or even entire careers. Some of these stories are truly egregious, like that of Dutch sociologist Diederick Stapel, whose falsifications ramified down into the careers of his graduate students too. This sort of extreme case is fortunately rare, but what it points to is the possibility to get away with even a massive fraud if no one ever checks on the work. Replicating scientific studies to verify the findings of other researchers is not as common as it ought to be, and the tendency of scientific journals to preferentially publish positive results put pressure on scientists to generate the right kind of study. Most scientists, and most citizen scientists, resist that tug, but we also do whatever we can on our end to make our program as rigorous as possible. As the editorial mentions, volunteers may not be expert in identifying, say, plants. Or dead birds. We want anyone to be able to volunteer with us, so we apply to everyone equally the requirement to take a photo of every dead bird found, whether the volunteer is a total newbie to birding, or a wildlife biologist. We reduce the impact of inter-observer variation in bird identification by having one person (me) do all photo verification. And when I’m not sure of an i.d., I send it out to you, my crack team of bird identifying experts. Sometimes, science requires relinquishing some ego. But the better job we do, the more our data will stand up to rigorous inspection, and the more the profile of you, the army of citizen scientists, will rise in the world. In defiance of those who would sniff.




Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: