What criteria are used to judge how well an editor has been doing?

How much do quantitative data (eg, readership numbers, Web hits) matter relative to qualitative data (eg, subjective ratings, strength and variety of journal content)?

Is performance judged purely on editorial grounds, or do business considerations also enter in (eg, journal revenue, editorial costs)?

Are evaluations mostly "summative" (ie, looking at past performance, mainly to make salary and contract decisions), or "formative" (ie, to provide guidance, looking forward, on how things might be done better), or both?

Is evaluation a formal (eg, structured, written) or an informal (eg, discussion) process, or both?

Whose responsibility is it to do the evaluation?

Any information along these lines would be much appreciated.

Frank Davidoff
_______________________________
I don't know what others will say, but I cannot imagine any reply making sense other than 'it all depends'…on the type of journal, ownership, audience, involvement of societies, etc.

IB Pless
Editor, Injury Prevention
_______________________________
Of course "it" will all depend—but I'm at least as much interested in the diversity of evaluation systems as common features. The problem is that so far I haven't been able to find out much of anything about how editors get evaluated, so I'm beginning to wonder whether they actually do get evaluated in any useful way(s).

Frank
_______________________________
This is a very interesting question or set of questions. Obviously, editors do not judge themselves. I forwarded this question to the chair of our Committee on Journal Publications (CJP), who replied:

"Good questions – I think this is something we could discuss at a future CJP meeting, as we really have no formal structure for it. I think it is a mixture of quantitative and qualitative factors – readership, submissions, Impact factor, complaints (!!). The relatively frequent turnover of editors, and the long lag between events and measurement – makes it quite difficult I think to ascribe any effects to an editor until at least half way through their term!

So, we may have an answer for you in another few months when we have in fact discussed the issue.

Michael D. Lockshin
Editor-in-Chief, Arthritis & Rheumatism
_______________________________
Surely, as with other jobs, evaluation is done by whether the editor achieves the goals/objectives agreed with the owner/employer?

Tim Albert
_______________________________
That's a bit simplistic, is it not? Recall the CMAJ fiasco. Is the editor supposed to please the employer exclusively, or actually do good work? Hopefully, the two objectives are in accord most of the time. But what about when they are not?

Vance
_______________________________
I recommend Richard Smith’s “The Trouble with Medical Journals” (Royal Society of Medicine Press 2006) Chapter 12 (Editorial misconduct, freedom and accountability: amateurs at work) to those interested in editor evaluation.

Bob Dellavalle
Dermatology Section Editor, UpToDate
_______________________________
I think the best criteria are the famous databases such as Thomson ISI and PubMed in which the journal is indexed or is going to be indexed. If the journal is currently indexed, its maintenance and promoting the Impact Factor is of high importance; if not, acceptance in these high-level databases could be a great norm. Of course, indexing and maintenance the indexing in excellent position demands a collective collaboration, which must be managed by the editor. So, in my opinion, a high score must be devoted to this issue.

MB Rokni
Associate Editor, Iranian J Publ Health
_______________________________
On the larger issue of how to resolve the tangled relationship between sponsors [pharma as well as medical devices], journal publishing and academia, see a recent editorial:

Academia, Journal Publishing and the Bio-Medical Industry

http://www.msmonographs.org/article.asp?issn=0973-1229;year=2007;volume=5;issue=1;spage=11;epage=14;aulast=Singh

Ajai Singh
_______________________________
I am Mustafa Afifi, Ministry of Health, Oman, and Saudi Medical J editorial member.

I wonder how could we install a continuous monitoring system for the quality of both editors and reviewers and not just evaluation of their outcome indicators?

Mustafa Afifi