ADVERTISEMENT
  About the SA Blog Network













Information Culture

Information Culture


Thoughts and analysis related to science information, data, publication and culture.
Information Culture HomeAboutContact

Altmetrics: emphasizing the plural

The views expressed are those of the author and are not necessarily those of Scientific American.


Email   PrintPrint



One of the challenges we face when using alternative metrics is the interpretation of what we measure.  This is even more confusing than interpreting traditional citation impact (which is challenging and confusing in itself) because “altmetrics” is an umbrella term for a wide range of activities. One can’t compare an article being bookmarked with it being reviewed by an expert, even though they are both non-traditional metrics.

Taylor (2013) classified altmetric activity into five levels of engagement: There is the social activity level, that takes place in general social media sites and is short and rapid (e.g.  “likes”), component (e.g. data) re-use, scholarly commentary (in science blogs, F1000 reviews, etc.), scholarly activity in academic platforms (e.g. Mendeley bookmarking) and mass media coverage.

In a new article “Party papers or policy discussions: an examination of highly shared papers using altmetric data,” Taylor and Plume (2014), used data from Altmetric.com, which tracks four of the five categories (social activity, scholarly commentary, scholarly activity and mass media. I think there are at least three other articles derived from their data). They collected data from the Altmetric.com API for four months, until January 17th, 2014. They collected 13,793 scholarly articles with at least one altmetric mention. Then, they looked at the 0.5% articles (69 overall) that received the most attention at the social activity category (tweets, likes, etc.). Only 8 of the 69 articles are full-fledged original research articles. The eight research articles have appealing titles (e.g. Climate change: vast costs of arctic change) and come from elite journals (e.g. Nature).

The other 61 documents are news items/features from prestigious journals, and when I say “prestigious journals” I mean “Nature.” Almost all of them are either Nature News or Nature News Feature items. After I scratched my head a few times, I tweeted to Mike Taylor, asking what happened to the rest of the journals. Euan Adie, founder of altmetric.com, saw the Twitter conversation between Mr. Taylor and me and solved the mystery: Science, for example, is a major source for research news but doesn’t give its news items Digital Object Identifiers, so altmetric.com can’t collect their data. Nature, however, does give their news item DOIs, hence their altmetric dominance. This is a good reminder that alternative metrics (and traditional ones as well) are only as good as their data.

When Taylor and Plume analyzed the other altmetric categories they found that only two articles appeared in all of them. One is a Nature article called “Cerebral organoids model human brain development and microcephaly,” which describes the creation of human brain models in cell culture and modeling brain disorders. The second is a PNAS article about predicting a person’s personal traits (e.g. sexual orientation, use of addictive substances) based on her or his Facebook “likes,” (“Private traits and attributes are predictable from digital records of human behavior”). The biggest overlap was between mass media coverage and scholarly commentary (31 articles out of the 69, see Figure 1). This fits with results from my research that showed that most NEJM articles covered by blog posts aggregated in ResearchBlogging.org were also covered by the New York Times, the news agency Reuters, or both.

Co-occurrence counts of articles comprising the top 0.5% of articles in each class. Source: Taylor & Plume, 2014

Obvious limitations of the research are its short period of data collection (four months) and the relatively small number of articles. We’ve already talked about the biggest limitation – articles without DOI can’t be taken into account. In general, this isn’t a problem for research articles and reviews, because they usually have a DOI, but can hurt unpublished conference proceedings and other documents without DOI. The altmetric data collected and used in this study is skewed (normal for this kind of studies), with 15% of the articles receiving about 90% of the social activity. The authors suggested wisely that in the future, document types (e.g. editorial, review, research article) should be taken into account when analyzing altmetric data, which is how it’s done in traditional citation analysis.  Overall, this is an interesting exercise in what altmetric data can and can’t do for us.

 

Taylor, M., & Plume, A. (2014). Party papers or policy discussions: an examination of highly shared papers using altmetric data Research Trends (36), 17-20

Taylor, M. (2013). Towards a common model of citation: some thoughts on merging altmetrics and bibliometrics Research Trends

Hadas Shema About the Author: Hadas Shema is an Information Science graduate student at Bar-Ilan University, Israel. She studies the characteristics of online scientific discourse and is a member of the European Union’s Academic Careers Understood through Measurement and Norms (ACUMEN) project. Hadas tweets at @Hadas_Shema.

The views expressed are those of the author and are not necessarily those of Scientific American.





Rights & Permissions

Comments 4 Comments

Add Comment
  1. 1. David Colquhoun 7:34 am 03/5/2014

    I’m afraid that this article does not deal adequately with the potential of altmetrics to do real harm to science. Altmetrics emphasize the trivial (and usually unread) papers at the expense of real hard science. They should be ignored totally.

    In order to see the harm you need to consider which papers score highly. Most bibliometricians don’t know enough about the subjects to judge. For a view from scientists who do, see http://www.dcscience.net/?p=6369

    Link to this
  2. 2. Hadas Shema in reply to Hadas Shema 10:02 am 03/5/2014

    Altmetrics, like any other indices, should be used with caution. And yes, we still don’t know enough, but that’s the way things were in the sixties with traditional citations – people started using them as evaluation tools before they were understood. It’s not an ideal situation, but altmetrics answer a need of the research community to go beyond citations.

    Link to this
  3. 3. JoshNicholson 10:41 am 03/5/2014

    We will begin publishing a series called The Grain and The Chaff that features essays by authors with famous and infamous papers.

    It is relevant here because The Grain will feature essays by authors with publications that have one thousand citations or more or a very high altmetric score (the top 250). When acquiring this list we noticed the same thing you did (many are news articles from Nature). Our aim in this series is to open up dialog beyond the metrics. If papers are so highly talked about perhaps the authors should be included in this discussion. You can read more here: http://thewinnower.com/blog/grain-chaff/

    We already have quite a few submitted and will begin publishing them at the end of the month.

    David, we would be happy to publish your piece to further open up discussion about altmetrics. If you’re interested please email us at contact at the winnower.

    Link to this
  4. 4. David Colquhoun 7:20 am 03/24/2014

    @Hadas Shema
    I have news for you. In the sixties nobody had heard of citation counting or any of the other manifold forms of bean-counting that now plague science.

    I have no ides what you mean by saying metrics should be “used with caution”, and I suspect that you have no idea either.

    Did you read the link that I gave? If so, what do you think about it.

    Altmetrics is a commercial venture, kids with computers trying to make a buck, but in the process doing real harm to science. I guess they haven’t the brains to do something more useful (like science). That’s not their fault but I just wish they’d get off the backs of the people who are trying to contribute.

    Link to this

Add a Comment
You must sign in or register as a ScientificAmerican.com member to submit a comment.

More from Scientific American

Scientific American Holiday Sale

Black Friday/Cyber Monday Blow-Out Sale

Enter code:
HOLIDAY 2014
at checkout

Get 20% off now! >

X

Email this Article

X