Case consultation: Authors do not respond to critical letters to the editor
A case was referred to the committee from a country that has large financial burdens and an insufficient budget for heath care. A specialty journal in that country reviewed and then published a clinical trial of an agent to treat a severe form of pulmonary disease. The trial concluded that a medication made in that country at greatly reduced cost was as effective as commercial products from other parts of the world that cost much more. It was accompanied by an editorial by a national expert commenting favorably on the findings.
Several letters of criticism were submitted after the publication of the article. They pointed out various weaknesses of design (no blinding, only patients in the "new drug" arm of the trial were asked for consent, very debatable and unusual choice of entry criteria, randomization unclear, etc.) as well as possible internal contradictions in some of the data and possible inconsistency with separately obtained data from the same parts of the country. In the view of the committee, none of the criticisms of data accuracy and inconsistency were specific or detailed enough to assess in terms of credibility, nor were the methods described in the original article detailed enough to make comparison easy. The author of the editorial has responded to these criticisms. The authors of the study (which was apparently funded by the government) did also, but only in the broadest fashion. Again, in the view of the committee member who reviewed the correspondence, their rebuttal was not very well organized or substantiated, but in all fairness neither were the criticisms that were submitted.
The editors of the journal were concerned that "there are high possibilities of fraud", but no such allegation was made in the letters except one that hinted that the unusual choice of enrollment criteria was used to make it more difficult to check on the authenticity of the data by other means. Some dramatic language was used in the correspondence, but the native speaker who translated the articles and correspondence stated that this is customary in this culture. All the other issues are those of poor study design and interpretation, which seem to have some merit. From the limited amount we know at present, we would characterize this as a flawed study with a number of major methodologic weaknesses, but with no clearcut allegation or evidence of "fraud".
The editors wanted to know what to do next. The following list of issues to consider was submitted to the committee:
- Publishing just the critical letters
- Recruiting and also publishing a critical editorial (to balance the previous favorable one)
After review the committee felt that this situation was best dealt with by simply publishing the letters of criticism and the author's and editorialist's response, as is common practice. If necessary, some of the inflammatory language could be edited out of the letters without distorting their arguments. No additional editorial is warranted at this time. The correspondence might not resolve the issue and might leave readers wondering as to what degree they could trust the original article, but that situation also is hardly novel.
The editors might also want to review the original peer review of the article, to see whether these issues were missed by their reviewers and whether this was a significant omission. This should help insure journal quality and might identify reviewers in need of more supervision, or perhaps even replacement.
If the editors felt for some reason that the authors' reply was deliberately evasive on some substantive issue (a point the committee could not determine based on the materials at hand), they might make an inquiry and a request for a more detailed response, initially to the authors and if necessary to the authors' institution (hospital or university). The paper notes that funding for the study was obtained from the federal government, which no doubt has rules about access to data and complete response to challenges by authors; if necessary the granting agency could be contacted to supply the needed information.