We’ve been quiet so far on a topic that’s been polarizing the blogs of late - the Canadian Blogging Awards. The Bunker’s reticence has a few reasons. The folks organizing the Awards are all nice people and good bloggers trying to do something fun. The folks taking issue with some aspects of the Awards are all nice people and good bloggers raising some valid concerns. Unfortunately the whole controversy is playing out in a medium where the norm is acrimony and red-hot rhetorical gamesmanship. These are bloggers, arguing about blogging, for crissake - It’s a given that ANY disagreement is going to assume the proportions of a global crisis, and any attempt at compromise on anything is going to be scorned as a shameful and unacceptable capitulation.

So we’ve just kept our mouths shut and let a bunch of folks we like and respect thrash it out.

But speaking as a sometimes professional evaluator of stuff, I thought I would add one comment about the notion of selecting the “best” of any category of blog.

A brief digression for illustrative purposes. Ottawa’s free arts and culture tabloid, the Ottawa Express, runs an annual poll to establish “the Best” in a number of categories. For a number of years, it decreed that Ottawa’s “best” hamburgers are to be found at Harvey’s, at the Rideau Shopping Centre.

Now, this always raised, to me, a couple of questions. What distinguishes these hamburgers from the other, identical, hamburgers served at the Harvey’s six blocks away on Sparks St.? Or at any of the Harvey’s establishments in Ottawa? Or Canada? And precisely what was it about these Harveyburgers that made them superior to the burgers served at, let’s say, “The Works”? Certainly not the quality of the meat, the bread, the service, the variety, the accompaniments, the atmosphere, the size…in all these particulars Harvey’s was, by any measure, inferior. The price? Maybe (compared to “The Works”), but the category was “best”, not “cheapest”. What, then?

A bit of reflection, of course, shows that the Ottawa Express poll is actually not identifying the “best” burger in Ottawa. It is identifying “The Hamburger Which the Largest Number of Readers of The Ottawa Express Will Identify As ‘Best’ Without Any Definition of The Term Or Selection Criteria”. That’s not exactly the same as “Best”.

To actually determine the “Best” of anything, a few basic steps are required.

1) You first have to decide what “best” means. What criterion, or combination of criteria, will be used to determine the “best”? Measured by some criteria, MacDonald’s is the “Best” restaurant in the world - measured by others, it falls somewhat short of Beckta’s or L’Atelier de Joël Robuchon. “Small Dead Animals”, the MacDonald’s of the Canadian Conservative blogosphere, is widely read, has a fiercely loyal following, and can mobilize its network to vote repeatedly. That may reflect some folks’ notion of “best”, but not everyone’s.

2) You have to decide how to define your sample. It’s entirely possible that the best burger in Ottawa is being made as we speak by my aunt Bernice for my cousin’s lunch. However, that falls outside the range admitted for consideration by Ottawa Express, which is “top X number of burger joints nominated by readers of the Ottawa Express”…a fairly limited sample.

3) You have to decide how to apply your criteria fairly, objectively and consistently to your sample. One way to do that is to define the criteria, and ask people to vote based on your criteria: but of course you have no way of knowing the extent to which they’ll agree. Another way is to define objective indicators for your criteria, collect the data, and analyze it.

So, if you REALLY wanted to determine the best hamburger within a defined sample, you’d:
- define evaluation criteria (which might include price, flavour on a scale, variety of choices, quality of meat, etc.)
- determine the importance of each criterion (does price matter as much as quality of meat?)
- define the indicators that will allow you to rate each sample under each of your criteria
- assess the samples and multiply each sample’s score in each criterion by its importance
- compare your findings.

That still doesn’t create a universal “best”. But it does give you a result that has a meaning…as opposed to a result that simply reflects the application of undefined criteria by an undefined group of respondents within an undefined sample.

Most current blog awards simply illustrate which blogs can mobilize a large number of voters. The assumption underlying that premise, I guess, is that an excellent blog will motivate people to do so more than a bad one. But that ain’t so. SDA and Raymi undoubtedly have their strengths - but it would be hard to suggest that they’re better written, better informed, or more supportive of intelligent discussion than other sites that fare less well. If your personal criteria for “best” include a high level of mindless acrimony or occasional boobs, fair enough - but those criteria may not be universally shared.

Awards that DON’T define selection criteria are still fun, and it’s always neat to get a nomination or a win. But we need to remember what’s being measured here - not merit, not quality, but the capacity to mobilize a network.

Recommend this Post at Progressive Bloggers

Check the community forum for related posts