On the reliability of Wikipedia

21 October 2007 | Community, Culture, Internet | Comments

A reporter at the USC college newspaper, the Daily Trojan, reached out to ask my opinion about a recent story in the Chronicle of Higher Education that showed that somebody with a USC IP address was changing the Wikipedia entry about the school. I found myself spending a few minutes writing out an email in response, and since — given my recent experience with MarketWatch — I doubt that more than 5% of what I wrote will wind up being used, I’ve decided to share the whole of it here.

The most interesting thing about the Wikipedia entry about USC being changed by somebody working from a USC IP address is that anybody noticed. And the second-most-interesting thing about this is that it wasn’t a person that noticed but an algorithm: the WikiScanner by Virgil Griffith at Cal Tech.

If the point of covering this story for the Daily Trojan is to alarm readers into thinking that the internet is not a reliable source of
information, then I don’t think that the WikiScanner/USC/Wikipedia incident is good evidence for that point of view. Quite to the contrary, the combo platter of WikiScanner and the Chronicle of Higher Education show that crowdsourcing information — that is, letting a vague and oosely arranged network bear the brunt of fact checking and exposing where information comes from — is a good mode of quality control. The system, in this case, worked. It doesn’t always work, but it did here.

There are two key things to remember about reference information and indeed all information:

The First Thing: it is always inherently biased and partial. You can never trust any single source to give you objective and unbiased information. In fact, you’ve NEVER been able to do that… But when there were only a few sources of information that was a lot harder to notice. There is not now nor has there ever been completely reliable information.

The Second Thing: any consensus by a large group of people that something is accurate or objectively true is always a consensus that has an expiration date… Although that expiration date is rarely well defined. Remember, it used to be that a majority of people in this country accepted as fact that it was OK to own other people so long as the color of their skin was different. For a more recent example, between the 1940s and the 1990s nobody questioned the factual accuracy of what some call the Shoah and others call the Holocaust, until a group of anti-semitic fanatics (David Irving is the most famous of these loons) decided that it was OK to perpetuate bad history and pernicious argumentation in favor of a political agenda.

It used to be that encyclopedias were created by groups of professional intellectuals who selected each other to create the information in the books. These professionals went to school for many years in order to be indoctrinated into an academic culture with norms and best practices that made them intelligible to each other and qualified them in each other’s eyes to create encylopedias. That still happens nowadays with encyclopedias like Britannica. However, network-created and curated encyclopedias like Wikipedia are also available for reference.

It’s simply not true that just anybody can change a Wikipedia entry willy-nilly. Anybody can TRY, but there’s an active and passionate group of volunteer Wikipedia editors that are the gatekeepers of the information and carefully scrutinize changes… Just like the professional editors at conventional encyclopedias.

The interesting thing is that for basic reference purposes, Wikipedia and Britannica aren’t all that different. You might take a look back at 2005 article in the journal “Nature” that found that pound for pound Wikipedia was generally as accurate as Britannica. The Britannica people just hated hearing that and have been quarelling with the study ever since, not than any Wikipedia users have noticed.

Here’s a link to an article from that time.

But please bear in mind that a vigorous debate on this has continued since then.

My final point: whether you’re doing research online or offline, trusting only one source of information is a mistake. Undergraduates today and internet users everywhere have developed a bias towards online information because it is so much more convenient than trundling off to library somewhere and wandering through the stacks or scrolling through endless reels of microfilm or infinite stacks of microfiche.

That bias means that digital information becomes more authoritative by default because it is the easiest to get, but the hard work of figuring out whether or not you trust that information, and where the biases in it lie, is still a burden on the researcher.

Wikipedia is “good enough” information for a casual query, and because it has become the default reference work for everybody online, its accuracy can be hotly contested and should be.

Frank Zappa allegedly said that “A mind is like a parachute. It doesn’t work if it’s not open.”

Open minds search beyond Wikipedia.