As the sole science librarian at a small liberal arts college, I work with faculty and students in a variety of disciplines. This means that I need to understand the literature of those disciplines, and understanding the literature means knowing at least a little bit about the metrics that are used to measure it: impact factors, h-indexes and altmetrics can all be interesting and useful, but establishing context can be difficult.
For example, is an h-index of 9 good, bad or indifferent?
Likewise, it is hard to evaluate impact factors across disciplines (along with many other problems with the IF). The flagship journal of the American Chemical Society (JACS) has an impact factor of 9.707. The flagship journal of the Geological Society of America (GSA Bulletin) has an impact factor of 3.787. We can’t compare the two like this, and we certainly can’t use these numbers to compare researchers from the two disciplines.
In a recent article in EMBO Reports, Bornmann and Marx (2013) argue for the greater use of percentiles in evaluating researchers, institutions and publications. You remember percentiles, right? When you took the SAT or the GRE, your results came back with a score and a percentile: if you were at the 85 percentile, you scored higher than 85% of your peers.
Percentiles can provide important context in one easy to read number. You know that the top is 100 and the bottom is 0. The median is 50, and this helps us makes sense of things even if the data set is skewed. However, it becomes incredibly important to select the right group to compare against. Typically, subject and year can create reasonable groups.
The folks who look at research metrics outside of citations also have the challenge of providing context. Altmetrics examine the different ways that folks interact with journal articles (in addition to citations). Are they talking about the article on Twitter? Saving the article to a bookmarking cite like CiteULike or Mendeley? Is the public citing the article on Wikipedia?
Once again, context is vital. An article was tweeted about twice. Is this good, bad or indifferent? Sixty folks on Mendeley have added it to their libraries. But what does that mean? The premier tool for easily showing and displaying altmetrics, Impact Story, can provide a bit of context for these numbers by calculating percentiles based on a comparison group of randomly selected items from the same publication year. Right now, it doesn’t appear that Impact Story is taking advantage of subject categories (which is more difficult). As a result, articles in some disciplines would automatically have lower percentiles as an artifact of lower average citations in that discipline.
Using any metric to evaluate scientific research is tricky – you are trying to boil down the intellectually complicated act of advancing human knowledge into a single number. But these metrics are being used more and more by tenure and promotion committees, institutional advisory boards, grant review committees and more. If folks choose to use metrics (like percentiles) that can provide reasonable and reliable context, we can avoid at least a couple of the standard pitfalls.