This month, DFTB launched our one-year trial of a new way of measuring impact and encouraging the referencing and sharing of our posts. We have done three things: added a Cite box, got DOIs for each post, and added the use of Altmetric.
A debate has been ongoing for several years about how best to measure impact and quality for FOAM resources. To date, there have been a few proposals.
Evaluation of quality in FOAM
The AIR modules are a project by the team at ALiEM to put together high-quality FOAM articles to make up a learning module on a particular topic. They have an Executive Board that reviews FOAM posts and scores them using an objective scoring system. The highest-scoring posts are included in the teaching module. Their scoring tool looks at BEEM scores, content accuracy, educational utility, evidence-based medicine, and references.
This series has been an excellent contribution to the conversation around FOAM evaluation. However, articles have to be selected to appear in an AIR module, so for those of us who would like to demonstrate the quality of each of our own posts, we need to consider another solution.
Evaluation of impact in FOAM
To measure impact, Brent Thoma started the social media index a few years ago, which ranks all FOAM sites for their social impact (DFTB is currently 15th overall, which, as a paediatric-specific blog, we are very happy with, although we always strive to improve).
There is a difference between quality and impact – this issue is faced by mainstream journal publications, too. To ensure quality, they have pre-publication peer review – we do a version of this for DFTB. To measure impact, they have Impact Factor, and many use Altmetric. They measure impact by how their articles are cited or shared by others. This does not take into account knowledge translation.
In the Weingart paper, the idea that an Impact Factor only centres on being cited by another article is discussed. The authors suggest that to assess the impact of FOAM, ‘a new Estimate of Clinical Influence could take into account the number of online page views an article receives, downloads, discussion forum comments, as well as mentions, links and viral sharing on Twitter, Facebook, and other social media.‘
In essence, this is what we already do with FOAM when we document how many times our article is shared. At DFTB, we have had sharing counts at the top of each post for many years, and we feel this transparency is important for readers, too. But sharing counts alone doesn’t resonate with mainstream publications, and we wanted to look for something that could compare us directly.
Why do we care about mainstream journals?
Peer-reviewed publications producing original research will always be the central tenet of clinical and scientific advancement. FOAM content cannot replace this, but it can be part of the conversation. There has been discussion previously about the role FOAM plays in knowledge translation and adding the voice of practical experience to allow the interpretation of scientific research and its integration into our clinical practice.
We have something to add to traditional journals, as our reach and ability to connect with the medical community can be a challenge for journals. Examples for DFTB are where we have worked with research groups to share our interpretation of their publications (e.g., the PREVIEW trial or the PARIS trial).
Many journals (including JAMA, EMJ, BMJ, and ADC) use Altmetric to assess broader social impact. We believe this is the integration of social media into mainstream journals predicted in the Weingart and Faust paper. To try to find common ground with mainstream journals regarding impact assessment, we decided to implement a one-year trial of Altmetric in DFTB.
Trial of Altmetric and Journal Comparison
We believe we are the first FOAM site to implement Altmetric, and the cost is the most likely barrier to this. Most FOAM sites (including DFTB) get no external funding and cannot afford the Altmetric fee. The DFTB team has been running an acute paediatric conference for the last two years (see DFTB18.com) and has been able to use funds from this to support us in continuing to produce FOAM content on the main DFTB website. This year, we have used the funds to purchase an Altmetric annual subscription.
Altmetric only works prospectively on posts, so although it is shown in older posts, it will only count shares from when we started our Altmetric subscription. We noted at the beginning that it was only counting tweets and missing Facebook, LinkedIn, and other blog shares, so there have been some teething issues, and it has taken a while to get up and running.
At the end of the year, we will aim to look back at the most recent posts and compare the Altmetrics to recent publications from mainstream journals. We can also compare our most popular posts to their most popular articles. Our aim is to use this to demonstrate the impact of our FOAM content and to speak in a language understood by mainstream publications.
Why do we have DOIs and a Cite box?
We now have DOIs for all DFTB posts. A DOI is a unique identifier that links to the post – you will see it in all journal articles. This facilitates easier citation of our posts and allows the tracking required for Altmetric scores.
Additionally, we’ve added a cite box inspired by First10EM, who implemented this some time back (EDIT: note that Scott Weingart at EMCrit first implemented this). This is simply a box at the top of each post stating how to reference our posts. This is the secondary aim of our project – to encourage people to cite our posts when writing articles for mainstream publications by putting them in an easy format that correlates with the requirements of journal citations.
References
Alper Cevik A, Aksel G, Akoglu H, Emre Eroglu S, Ozgur Dogan N, Ali Altunci Y. Social media, FOAMed in medical education and knowledge sharing: Local experiences with international perspective. Turkish Journal of Emergency Medicine. 2016;16(3);112-117. https://doi.org/10.1016/j.tjem.2016.07.001.
Grock A, Paolo W. Free open access medical education: a critical appraisal of techniques for quality assessment and content discovery. Clinical and Experimental Emergency Medicine. 2016;3(3):183-185. doi:10.15441/ceem.16.156.
Weingart S, Faust J. Future evolution of traditional journals and social media medical education. Em Med Aus. 2914;26:62-66. doi: 10.1111/1742-6723.12192.