DOI and Altmetric

This month, DFTB launched our one year trial of a new way of measuring impact and encouraging the referencing and sharing of our posts. We have done three things: added a Cite box; got DOIs for each post; added the use of Altmetric.

Debate has been ongoing for several years about how best to measure impact and quality for FOAM resources. There have been a few proposals to date.

Evaluation of quality in FOAM

The AIR modules are a project by the team at ALiEM to put together high quality FOAM articles to make up a learning module on a particular topic. They have an Executive Board which reviews FOAM posts and scores them using an objective scoring system. The highest scoring posts are included in the teaching module. Their scoring tool looks at: BEEM score; content accuracy; educational utility; evidence based medicine and references.

This series has been an excellent contribution to the conversation around FOAM evaluation. However, articles have to be selected to appear in an AIR module, so for those us who would like to be able to demonstrate the quality of each of own posts, we need to consider another solution.

Evaluation of impact in FOAM

To measure impact, Brent Thoma started the social media index a few years ago which ranks all FOAM sites for their social impact (DFTB is currently 15th overall which as a paediatric specific blog we are very happy with indeed, although always strive to improve).

There is a difference between quality and impact – this is an issue faced by mainstream journal publications too. To ensure quality, they have pre-publication peer review – we do a version of this for DFTB. To measure impact, they have Impact Factor and many use Altmetric. They measure impact by how their articles are cited or shared by others. This does not take into account knowledge translation.

In the Weingart paper there is discussion around the idea that Impact Factor only centres on being cited by another article. The authors suggest that to assess the impact of FOAM ‘a new Estimate of Clinical Influence could take into account number of online page views an article receives, downloads, discussion forum comments, as well as mentions, links and viral sharing on Twitter, Facebook, and other social media.‘

In essence this is what we do already with FOAM when we document how many times our article is shared. At DFTB we have had sharing counts at the top of each post for many years and we feel this transparency is important for readers too. But sharing counts alone doesn’t resonate with mainstream publications, and we wanted to look for something that could compare us directly.

Why do we care about mainstream journals?

Peer-reviewed publications producing original research will always be the main tenet of clinical and scientific advancement. FOAM content cannot replace this, but it can be part of the conversation. There has been discussion previously about the role FOAM has to play in knowledge translation, and adding the voice of practical experience to allow the interpretation of scientific research and its integration into our clinical practice.

We have something to add to traditional journals, as our reach and ability to connect with the medical community is something that can be a challenge for journals. Examples for DFTB are where we have worked with research groups to share our interpretation of their publications (i.e. the PREVIEW trial or the PARIS trial).

Altmetric is used by many journals (including JAMA, EMJ, BMJ, and ADC) for the purpose assessing broader social impact, and we believe this is the integration of social media into mainstream journals that was predicted in the Weingart and Faust paper. In order to try to find common ground with mainstream journals regarding impact assessment, we decided to implement a one year trial of Altmetric in DFTB.

Trial of Altmetric and journal comparison

We believe we are the first FOAM site to implement Altmetric and a barrier to this is most likely the cost. Most FOAM sites (including DFTB) get no external funding, and therefore cannot afford the Altmetric fee. The DFTB team has been running an acute paediatric conference for the last two years (see DFTB18.com), and have been able to use funds from this to support us continuing to produce FOAM content on the main DFTB website. This year we have used the funds to purchase an Altmetric annual subscription.

Altmetric only works prospectively on posts, so although it is shown in older posts, it will only count shares from when we started our Altmetric subscription. We noted at the beginning that it was only counting tweets and missing Facebook, LinkedIn, and other blogs shares, so there have been some teething issues and it has taken a while to get up and running.

At the end of the year we will aim to look back at the most recent posts and compare the Altmetrics to recent publications from mainstream journals. We can also compare our most popular posts to their most popular articles. Our aim is to be able to use this to demonstrate the impact of our FOAM content, and also to be able to speak in a language understood by mainstream publications.

Why do we have DOIs and a Cite box?

We now have DOIs for all DFTB posts. A DOI is a unique identifier that links to the post – you will have seen them on all journal articles. This facilitates easier citing of our posts, and also allows the tracking required for Altmetric scores.

Additionally, we’ve added a cite box, inspired by First10EM who implemented this some time back (EDIT: note that this was first implemented by Scott Weingart at EMCrit). This is simply a box at the top of each post stating how to reference our posts. This is the secondary aim of our project – to encourage people to cite our posts when writing articles for mainstream publications by putting them in an easy format that correlates with the requirements of journal citations.