Should a Specific Number of References Be Required?

The Pan-American Journal of Public Health (RPSP/PAJPH) requires that research articles contain a minimum of 20 bibliographic references (but also notes that these should be relevant and current).
 

I haven't seen this requirement in any other journal and would be interested to know:

(i) do any other journals ask for this?

(ii) is there any evidence that longer reference lists equate with higher quality reporting?

(iii) what do other editors think about the idea? 

Liz Wager
_______________________________
We have no such policy at our journal; in fact, we ask in our instructions that authors limit themselves to truly relevant references, as authors sometimes overdo it with their reference lists (for example, giving a citation, or several, for a very general statement like "The U.S. healthcare system has undergone many changes in recent years.")  We state in our Instructions for Authors: "the goal should be to present representative and illustrative references only, not a comprehensive list."  We certainly don't want authors to UNDER cite, but they should strike the appropriate balance.

Lisa Dittrich
_______________________________
Bizarre idea…totally unscientific.

I. B. Pless
Editor, Injury Prevention
_______________________________
When we see either an unusually long reference or author list, we check for "CV padding."

Kimberly Taylor
Publications Director, Journal of the National Medical Association
_______________________________
I am the editor of Colombia Medica, a Colombian medical journal. We are now in our 37th year of uninterrupted publication. We have the Vancouver style as requirement for all articles send for publishing. We asked for no more than 20 references for research papers. We think that is enough.

Guillermo Llanos
_______________________________
Liz Wager asked: “What do other editors think about this idea?”

This is what I think:

What is a reference list, after all?

It just shows the author/authors have read relevant material, are abreast of current/landmark work in the field, and are not speaking out of their hat. When a publication requests for 20 minimum references, it is making sure the concerned authors have at least gone through 20 other related works in the field. What's wrong worth that?

The only problem is when 20 people have not worked in a certain field still. Or if they have, are not worth citing. In such a case the concerned publication can waive the condition. I think the intention in putting this condition is laudable and worthy of implementation elsewhere too.

I have doubts whether most worthwhile publications have less than 20 references any way. (Except editorials, brief communications etc.). Making it mandatory may be one way of making the less than worthwhile come up to certain standards with regard to contemporaneity and relevance.

Of course, like every good idea, this can be twisted out of context. Quoting peripheral works, or self-citations, just to make up the magical number may occur. But that can happen even otherwise. So I can well understand why the publication may have come up with such an idea.

Ajai
_______________________________
We have a maximum of 5 references for single page and 10 for double (1400 words) but have no specific requirement for any references, as I don't believe they are always needed. I leave it up to the individual authors.

Vivienne Miller
Diabetes Management Journal
_______________________________
If your research article is a unique one you ought to collect at least 20 supporting references for this journal. If you couldn't find it, will you wait for it? Sometimes (as I came across once) you can find a lot references but all from an institute with different first author's names, published in different journals. Will these references increase the quality?

In my opinion, restrictions and/or forcing on the length of the reference list are not valid with the quality of the individual research article.

Nezih Oktar
JNS  (http://www.jns.dergisi.org)
_______________________________
Surely the length of a reference list depends on the content, which in turn depends on the aim of the paper.

I agree that ceilings (tall or short) are an absolute nonsense...at best, and at worst may hide manipulations.

Tom Jefferson
Cochrane Vaccines Field
_______________________________
Our journal doesn't deal with a lot primary research publications;  it is a journal of management. The limit on references is purely practical, as one author wanted 61 references for 1400 words and we only have 36 pages to publish!

Vivienne Miller
_______________________________
Thanks, Liz, for initiating a new discussion string. ICMJE does not have a clear policy in this regard, and I wish they could come up with one. We have developed our journal's policy that I am sharing with you all.

"Any number of references is acceptable as long as the following criteria are met with:

*Total article length including references is not more than 3000 words (upper limit recommended by ICMJE)

*References are relevant (determined by subject specialist/peer reviewers)

*References are latest possible (determined by subject specialist/peer reviewers/bibliographer)

*References can be authenticated (determined by bibliographer)

*No more than 3 references are given for a single point (determined by subject specialist/peer reviewers/bibliographer)”

I am sure we all will learn from each other from this discussion.

Ahmed Badar
Managing Editor, Journal of Ayub Medical College (JAMC)
_______________________________
I would also agree to this.

I think it should not be a number game. However, one might consider:

1) Relevance to the topic

2) Diversity

3) Recent references

All we need to know is that the author has gone through an exhaustive list of references and has the ability  to rationalize the point he is making in the light of referred experiences.

A rare topic might have limited number of references and it may not be considered as a drawback to the study. Moreover, it is always the art of expression that  makes a particular reference valid.

Fazal Ather
WHO Consultant, EMRO
Editorial Board, JPMI
_______________________________
Specifying a minimum of 20 references is in my view absurd, but so is specifying a maximum. The same applies to limits on the total word count, number of tables and figures, etc. It is part of the role of the peer reviewer to judge whether the paper is over- or under-referenced, etc.

All such limits are of course arbitrary, and have no scientific basis. They are also increasingly irrelevant as for most journals additional material can be put on the web. Blanket limits take no account of the variable nature of the scientific article and the need to communicate adequate information about a study. Editors should use judgment based on a study's importance, complexity, etc, not rely on unsupportable rules. It is fine, however, to say that exceeding a certain length would be unusual and that doing so may reduce the chances of publication, but you have to allow flexibility in relation to the nature of the article. To take just one simple example, an article reporting a systematic review should include the references of all the articles included - this could easily exceed 100.

Imposing inflexible limits encourages authors to publish articles that  exclude details vital to understand how the study was done or what it found. Those articles fail adequately to serve either the participants in the research or the consumers of the research report.

Just because it is much easier to specify cosmetic aspects such as word limits than aspects of scientific content does not mean that this easy option should be taken. Editors should worry much more about content and less about structure. A huge amount of published literature shows that a high proportion of published articles omit key information about the studies they are describing. This is a failure of the medical literature as a whole which WAME members should be very concerned about. Yet most medical journals focus almost entirely on formatting issues in their 'instructions to authors' and say very little if anything about scientific content.*

Sorry for drifting way past the original question but the larger picture is extremely important and, I believe, rather neglected by WAME.

Doug

* Schriger DL, Arora S, Altman DG. The Content of Medical Journal Instructions for Authors. In Press Corrected Proof, Available online 5 June 2006 Annals of Emergency Medicine DOI: 10.1016/j.annemergmed.2006.03.028
_______________________________
I couldn't agree more. Clarity is served neither by squeezing information into fewer words than are really necessary to convey the information, nor by sending readers to an appendix for key aspects of the study.

If saving paper and money is the object, journals could simply print abstracts and put complete, clearly written articles on the Web. 

Susan Eastwood
_______________________________
Doug Altman is right: there are some broader issues here.

One problem is the way that many people assemble their articles, which is by stringing together a whole range of references.

One useful technique I have found is to get them to 'free write' their article, then go through it sentence by sentence to see which ones need supporting by a reference. They can then put them in—and also go through their references to see if there are any important gaps, and revise accordingly. In this way, they are using references for their original purpose, which is to support statements in the text (and not to show that they have read the literature).

Incidentally, if there are any alternatives for a reference, as there often are, I suggest that they cite articles in their target journal. This will show that they are coming in on a thread (or conversation).

Tim Albert
_______________________________
Two issues deserve mention here. First, peer reviewers not uncommonly point out that the references authors cite don't support the statement they are intended to support. That may be a hazard of writing first what one believes and then finding a reference to support that belief; an evidence-based discussion should begin with thoroughly reading the literature and drawing conclusions from it, not vice versa. Some authors may be able to write first and cite later if they truly know all the relevant literature, but they still should be careful that their citations truly support what they have said.

Second, studies have shown that authors often do not actually read the entire reference they cite, and authors may propagate both misunderstandings about what an article actually stated as well as incorrect citations. Authors should be encouraged to read every reference they cite before submission (and ensure that the article has not been retracted, as stated in the ICMJE guidelines: http://www.icmje.org/index.html#manuscript

Margaret Winker
Deputy Editor, JAMA
_______________________________
However, one needs to take the nature of the journal into account when deciding on reference numbers (or not). For example, if it is a management journal (such as ours) with 36 pages and a specialty subject, it is not practical to have 61 references (2 times as much space as  the article itself). Some journals are best served by having a maximum number of references. I do not believe in forcing an author to put references if he/she doesn't want to, but I believe the standard of the article looks better with references.

Vivienne Miller
Diabetes Management Journal
_______________________________
I arrive a bit late to this highly interesting discussion initiated by Liz Wager regarding the legitimacy and appropriateness of requiring a minimum number of references for research articles, with Revista Panamericana de Salud Pública (RPSP), of which I happen to be the Managing Editor, being a case in point.

The requirement in question predates me, but I can sympathize with the Managing Editor before me who saw fit to establish it. There was a time when scientific writing from a few underdeveloped countries in the American Region was less sophisticated than it is today, and it was not unusual for research articles submitted to RPSP by authors from those countries to be almost devoid of references substantiating the remarks made in the Introduction, for example, or in the Discussion section. In an effort to educate authors as to the role and importance of crediting the sources of information or ideas that were not a matter of personal opinion only, it became rather necessary to address the issue by establishing a minimum number of references for scientific research papers. This is not to say that the 20-reference minimum is imposed as an inflexible requirement, nor is it an indication that the journal focuses exclusively on format rather than on content or scientific quality. In fact, it is not altogether clear to me why such a conclusion should be drawn. Every article that is received is treated on an individual basis, and it is obviously not the number of references that drives editorial decisions in the last analysis, but whether the references are current, relevant, and sufficient.

The quality of the research papers submitted to the journal has improved substantially over the years, largely as a result of courses, workshops, and tutorials on scientific writing in Latin America. It may be time to revisit the minimum reference requirement, which is obviously quite arbitrary and "unscientific," as some have pointed out. In fact, I take this opportunity to thank Liz for bringing up the issue and the rest of my colleagues for voicing their sincere opinions. This type of open dialogue is what makes this forum such an invaluable source of ideas for improving our editorial practices.

María L. Clark
Managing Editor, Revista Panamericana de Salud Pública/Pan American Journal of Public Health
_______________________________
I also guess that it would be interesting to hear from other editors whether in your experience you have had an author who has negotiated a word limit or reference limit, ie something like "I have seen the word and reference limit but I guess my manuscript would be better with 5 more references or 100 more words. If this has ever happened to you, what has been your response?

Adamson Muula
Malawi Medical Journal
_______________________________
I also agree with the freedom of the editors/publishers to determine  the length of papers, reviews, case reports, etc, as well as the list of references. Any paper on the serial "Annual Review of ..." is above the 100 references and that is alright. Some journals, editorials, or comments have only one or no references. That is alright, too. I have never seen a journal asking for a minimum length of reference list.

Normally, they ask for a maximum in order to avoid repetition, self-citing, obsolescence, ambiguity, etc. Our journal asks for a maximum of 25 references per paper; however, that is not compulsory and we let the authors and reviewers  agree on the relevant references to be printed. Our invited reviews normally have more than 60 references. That is alright for us, by now!

Pedro José Salinas
Editor-in-Chief, MedULA, Journal of the Faculty of Medicine of the Universidad de Los Andes
_______________________________
Interesting remark of Tim's about suggesting that authors "cite articles in their target journal. This will show that they are coming in on a thread (or conversation)," in light of yesterday's Wall Street Journal piece where various people had apoplexy about editors making the same suggestion. The orthodox take on this is that such a request is a naked attempt to pump an impact factor, but Tim's remark goes to something important that I think is under-discussed: journals often do have dedicated readerships who may not read widely beyond that journal. This would be the case with many clinical membership journals where the readers are not mainly researchers, but practitioners wanting to understand the progression of an idea or finding. Also, some specialist journals are the main repository of the best scholarship about particular issues. In both these cases, editors might want to encourage authors "to come in on a conversation" which has been developing in past issues for the benefit of readers keen to follow emerging developments. But if they were to ever suggest such a heinous thing, they would be lined up against a wall by the impact factor cheating stalkers.

I've noticed too that reviewers also quite often recommend that authors cite extra studies, which are often the reviewer's articles. If a paper has been sent to reviewers who have direct expertise/research experience in the subject of the paper (which is of course a very sensible thing to do), and authors have missed citing their work where this is appropriate, then to my mind this is another minor sin that I imagine would suffer the slings and arrows of innuendo about individuals pumping their own citation rates.

Simon Chapman
Tobacco Control
_______________________________
Individual negotiation and focus on content and not form seem like the traditional, right approach to me.

I'd only like to remind everyone that in negotiating content and form with authors individually, it's important to watch over the revision process. In this case, to see that cutting is done appropriately—but the same principle would be true when asking an author to add in references. There are many novice authors out there who don't know what's entailed by a reviewer's request. I've seen situations where reviewers ordered reference cuts and authors returned something more like an encyclopedia article, with huge swaths of text in need of those references! The editors were satisfied that the overall manuscript was shorter, but imagine the reaction of the copyeditor! I've also seen authors inserting requested references in wildly inappropriate places and no one noticing until it reached the copyeditor.

To echo what Doug Altman said: noticing mismatching between content and form is a reviewer's job. I think that watching for continued mismatches in a revised manuscript is an editor's job, with possible additional input from a reviewer.

M.E. Kerans
_______________________________
Dear Doug and others, 

1. I agree that enforcing a minimum number of references is complete nonsense. However, I do disagree with Doug about a maximum number.

2. It is entirely feasible and right to set a maximum number depending on the type of article. There is little worse in a print journal than page after page of references. Luckily, the web allows both Doug and I to be satisfied with fewer references in print and the full list on the web. 

3. I also agree with Tim and Simon that there is too much puritanism about not referencing your own journal. Clearly, this should not be done gratuitously or inappropriately but I do believe that journals should be providing a service to readers and part of that service is linking back to relevant articles in your own journal be that via the reference list or some other mechanism so that readers can follow the "story." Of course, the story might lead readers to other journals, but unfortunately all journals don't have the same access policies.

Kamran Abbasi
Editor, JRSM