June 21, 2014 | 2
My PhD mostly dealt with research blogs from ResearchBlogging.org (RB) an aggregator of blog posts covering peer-reviewed research. In this article, we (Prof. Judit Bar-Ilan, Prof. Mike Thelwall and I) describe studying bloggers’ motivations using a content analysis approach. That is, we read 10% of the RB “Health” category between 2010-2012, overall 391 posts. The result was an article called “How is research blogged? A content analysis approach” (green OA version here). At first, we attempted to content-analyze every citation, but that turned out to be extremely difficult – though bloggers cite at least one source the formal way, they don’t always cite all of them, and even when they do, they don’t always attribute every piece of information in the post to its source.
We could only roughly classify citations into the categories “confirmative,” “partly negational,” and “negational.” A citation was “confirmative” by default unless the blogger stated otherwise, because we assumed that a decision to an article without criticizing it was a way of endorsing its content. We usually did not consider the mention of a study’s limitations (e.g. a pilot study) as negational, since studies always have some limitations. If they were just mentioned in passing, the citation was still considered “confirmative.”
References were “partially negational” in cases in which the blogger rejected certain elements of the article but accepted others. For example:
But why look at the consumption of “social drugs” in people who are depressed? Aren’t these individuals less inclined to be social? And aren’t they likely to show anhedonia (loss of interest of pleasure) according to DSM IV criteria? (The Neurocritic, 2011).
The Neurocritic pointed out that the article is flawed, but hasn’t dismissed the article’s results altogether.
A “negational” citation, however, meant that bloggers printed the article just for the pleasure of burning it. Here’s a quote from Rogue Medic, expressing his very honest opinion about an article:
Survival to admission is a net beneﬁt? A lot more patients dying in the hospital is a net beneﬁt? Survival to discharge is a real beneﬁt. Imagine telling a patient’s family, “She died, but she lived long enough to acquire thousands of dollars in hospital bills. We consider this a net beneﬁt.” A surrogate endpoint is not a real beneﬁt!”
As fun as negational citations were, they were a rarity, as you can see in Table 1. This is in line with traditional citation content/context analysis studies, which showed, for example, 14% negational citations (Moravcsik and Murugesan, 1975) and 5% partly negational (none were completely negational) (Chubin and Moitra, 1975).
Table 1. Classiﬁcation of the extent of negation in the sampled blog references according to their post context.
We created a 10-category motivation classification, most with sub-categories, for the most popular motives we coded. All motivations could relate to issues concerning the general public, experts only, or both. For example, an ethical problem could be general (e.g., should dogs’ tails be docked?) or discipline speciﬁc (e.g., is it ethical to prescribe a placebo to patients?). A post could have more than one motivation (average was 2.8) hence the following percentages exceeding 100%.
I will focus on the categories I consider the most interesting: Criticism, Advice and Controversy.
The bloggers were interested in spreading helpful information; the “Advice” category consisted of “providing practical advice and recommendations” (in 25.6% of the posts) and “advocacy against certain treatment/life style/intervention” (in 2.0% of the posts). Seems that bloggers prefer to give advice than to argue against a practice, life style or treatment.
About 30% included some sort of criticism. This might seem odd, since I just mentioned that most references in academic articles and in our study are conﬁrmative, but the bloggers didn’t necessarily criticized the articles they covered. Yoni Freedhoff from Weighty Matters, for example, used an article to dispute the President of the Canadian Beverage Association claims regarding sugar.
“Mr. Sherwood is the President of the Canadian Beverage Association and just this week he was tasked with defending sugar as a contributor to obesity, diabetes and heart disease. Let’s review Mr. Sherwood’s letter, but seen through the lens of Kelly Brownell and Kenneth Warner’s Big Tobacco Playbook – whereby they came up with a list of plays that the food industry has co-opted from the early days of the ﬁght to prove tobacco harmful. (Freedhoff, 2012)”
Only 16 posts (4.1%) criticized an academic article’s conclusions or recommendations. According to Cole and Cole (1971) researchers usually ignore and do not bother citing works they think are low-quality. In order to be worthy of criticism, the work has to be influential. Cole and Cole’s conclusions seem in line with our findings.
The “Controversy” category could represent either an inner disciplinary controversy or a public one, and was found in 11.5% of the posts. One of the controversies discussed was that of food labelling:
“Recently, there has been a push to mandate labelling in fast food restaurants and stores. In the US, this is a huge initiative, passed as part of the 2010 Health Reform Bill” (De Winter, 2012)
Naturally, the study has many limitations; we only classified motives using content analysis, which means that it is our perspective of the bloggers’ motivations rather than necessarily their true motives; The results are based on 10% of a specific category in the RB aggregator and might not work for other discipline (some of the sub-categories related specifically to health).
The bloggers’ tendency to give advice might be especially significant, given that many members of the public search for health information online. As for criticism, we did find a fair amount of it, but mostly it wasn’t aimed at the articles covered in the post but at the media, common beliefs, etc. This could be because the bloggers, like academic authors, tend to cover articles they regard as valid. We also suggest this might have to do with the bloggers mostly posting under their own name and could be held accountable for criticism of, say, colleagues.
Blogs posts are already a source of alternative metrics (altmetrics), but to be accepted as a valid source of impact in the scientific community, they (and other altmetrics) must be better understood.
Chubin, D.E., & Moitra, S.D. (1975). Content analysis of references:Adjunct or alternative to citation counting? Social Studies of Science, 5,423–441.
Cole, J., & Cole, S. (1971). Measuring the quality of sociological research.Problems in the use of the Science Citation Index. American Sociologist,6(1), 23–29.
De Winter, G. (2012, June 22). Mutant flu, study two [Blog Post]. Retrieved from
Freedhoff, Y. (2012). Justin Sherwood, Refreshments Canada president,
defends sugar [blog post]. Retrieved from http://www.weightymatters.
Moravcsik, M., & Murugesan, P. (1975). Some results on the function andquality of citations. Social Studies of Science, 5, 86–92.
Noonan, T. (2010). Amiodarone for cardiac arrest in the 2010 ACLS—Part
III [blog post]. Retrieved from http://roguemedic.com/2010/12/
The Neurocritic (2011). Abusing chocolate and bipolar diagnoses [blog
post]. Retrieved from http://neurocritic.blogspot.co.il/2011/05/abusing-
Shema, H., Bar-Ilan, J., Thelwall, M. (2014). How is Research Blogged? A Content Analysis Approach JASIST : 10.1002/asi.23239