My thoughts about the problems of the IETF

[John C Klensin]

Keith wrote...

> however, if we give outside reviewers any formal power we
> need to define ethics for such review - such as, reviewers
> should not accept compensation for reviews.  otherwise people
> will try to buy favorable reviews for their pet projects and
> unfavorable reviews for others.

I've been thinking about this since you said it, especially in 
combination with

> from personal experience, adding outside reviewers can
> increase AD workload, because the AD needs to evaluate the
> reviews in addition to the documents under review.  (outside
> reviews can and do help increase document quality because they
> catch thing that the IESG and working group miss, but they
> don't generally reduce workload.)

earlier in the same note.

Suppose we use outside reviewers, whether chosen by the AD for a 
particular document, or chosen by the WG or some of its 
participants, or as part of a SIR-like pool.  Some of them will 
do careful and thoughtful review jobs, and will write reviews 
that identify and explain their concerns.  Others, if we can 
extrapolate from mailing list and other experience, will find 
design properties that match or conflict with their religious 
preferences and will produce reviews that mostly consist of 
ranting about those preferences.   And, extrapolating again, 
some will deliver pithy and clever, but almost uninterpretable, 
comments, whether those comments favor or attack the contents of 
the documents under review.

I suggest that we already have ample experience to infer that 
these different types of responses will occur, with or without 
specific compensation.  If an AD is going to make any use of 
outside reviews, he or she is going to need to sort these things 
out and reach conclusions about which reviews are credible, 
which ones are merely ranting, and which sets of abbreviated 
comments are worth following up to the extent needed to turn 
them into specific recommendations for changes.  And, yes, as 
you point out, that may add to the workload.  Some of that added 
load may lessen over time as the AD, and the IESG generally, 
figure out the correlations between particular individuals and 
untrustworthy or low-information reviews.

But I suggest that drawing the line at compensation or not is 
unlikely to be helpful and might be harmful.  Probably there are 
people in the community (although I have encountered few, if 
any, of them) from whom a favorable (or unfavorable) review can 
be purchased.  But that can possibly be done in other ways, 
including just picking reviewers who have some external interest 
in the approval or disapproval of a document.  By contrast, it 
seems to me that compensating someone -- in dollars, beer, kind 
words, promises to stop the pain of arm-twisting, or other 
currencies -- to pay attention to, and review, a document that 
he or she would not normally spend time looking at may not be 
totally unreasonable, at least as long as we permit people who 
draw salaries from companies with an interest in particular 
standards to act as WG Chairs, Editors, or even active 
participants.  And, as has been pointed out in other contexts, 
if we start banning the latter, we assure ourselves of documents 
that are produced by people with no real knowledge of or 
experience with, or involvement in, the subject matter.

So, looking at disclosure requirements --as long as they can be 
defined to prevent bureaucratic ridiculousness-- seems sensible 
to me.  But banning "compensation" --especially some particular 
form of compensation and not others-- does not.

       john