Staying on Track (Re: Documenting pilots (RE: pausable explanation for the Document Series))

John C Klensin john-ietf at
Sat Jun 7 11:01:34 CEST 2003


I need to _slightly_ support Melinda's position on this.  I 
certainly wouldn't have used words like "preemptive" or 
"uncooperative".  But...

In a cultural problem that goes back long before we, as you 
observed elsewhere, started putting more focus on static 
description than on demonstrated interoperability (and, IMO, 
blew it on both), we have had a habit of proposing, documenting, 
and sometimes even trying "experiments" which lack evaluation 
criteria... or any way, other than force of personality, to 
really determine whether they "worked".  Nor do we have real 
procedures for documenting what we have learned.

In that regard, SIRS has been, I think, better than the average. 
But it is a handy example, so let's look at it, with the 
understanding that I'm deliberately broadening the example a bit 
to illustrate a problem with the way we often do business ... 
some of that example may not actually apply to SIRS.

It establishes goals as:

>      The procedure described in this document is intended to
>      solve, or palliate, a number of related problems that
>      have been observed in the IETF process [PROBLEM]:
>           *    overload of the IESG (leading to delay)
>           *    perception that authority and influence
>                in the IETF are concentrated in too few
>                hands
>           *    submission of documents to the IESG that
>                still have significant problems (leading
>                to delay)
>           *    failure to detect fundamental problems
>                and Internet- wide issues at an early
>                stage

Certainly worthwhile.  But, quoting from your note, "...started 
offer itself for public use, and will prove useful -- or not -- 
on its merits.   How are those merits to be evaluated so as to 
determine whether this is a success?  The document then goes on 
to say:

>      However,
>      it is likely that drafts with at least three positive
>      reviews from SIRs in different areas will experience
>      much shorter IESG review cycles than drafts with fewer
>      positive reviews.
>      In all likelihood, Drafts without reviews will get
>      worse IESG response time than today, whereas Drafts
>      with reviews will be processed much more rapidly,
>      especially as the IESG's confidence in the SIR
>      procedure increases.

Plausible hypotheses indeed.  But, for a given document coming 
out of a given WG, we have no practical and objective way to 
accurately estimate how long it would have taken to get through 
the IESG without SIRS support, so it is not possible to 
objectively know whether SIRS sped thing up.

I don't think it likely, but a counter-hypothesis is that the 
IESG, upon receiving a document with three or five SIRS signoffs 
but disagreeing anyway, would decide that it needed to give more 
extensive explanations to those senior people, or discuss issues 
with them even before going back to discuss them with the WG. 
That would probably positively leverage quality, but would have 
a negative effect on speed.

And, of course, trying to guess at whether SIRS improved quality 
in comparison to what the IESG would have done, to/with a given 
document without it. is even more difficult.

So, you collect some reviewers, and some WGs choose to use them. 
For some of those WGs --perhaps the ones who were in good shape 
anyway but that, too, is pretty subjective-- the reviews help 
the IESG expedite the process.  For others, they don't appear 
to.  Do we then declare the experiment a success and try to make 
it mandatory?   Does anyone who doesn't think it has proven 
itself a success get to speak up without being overwhelmed in 
rhetoric and forceful assertions?  Do we infer that ADs who 
don't quickly sign off on SIRS-endorsed documents are 
obstructionist nit-pickers and need to be fired?   And so on.

As I hope my recent notes make clear, I'm not in favor of trying 
to invent procedures to alleviate any of the above, or of 
delaying things until such procedures reach some extraordinary 
consensus.  But I'd be much happier about the idea --even as an 
experiment-- had the IESG been willing to stand up and say 
"fascinating idea, lets try it".  And it would have been even 
better, at least from an "experimental" context, had the IESG 
chosen to say "It would be good to really do this as an 
experiment and therefore to be able to make some comparisons. 
So let's look at the following mix of areas, or specific groups 
of WGs, first, leave the others with status quo, and see how 
things work out.

I'm disappointed that type of response from the IESG hasn't 
happened.  Maybe they don't think they have been asked.   And 
the latter is, of course, what my recent notes have been about: 
if we need a formal document and request to publish it as an RFC 
and/or a formal WG call for consensus and response, in order to 
get an IESG response to a "we think this would be worth trying, 
do you have reactions or suggestions/requirements about how to 
try it", then, IMO, we are in deep trouble.    But, going ahead 
with something on the grounds that it is an experiment, without 
objective evaluation criteria or any real possibility of them, 
and without IESG comment, input, or participation...  That seems 
to me to be an opportunity for some future demagogue, with some 
other experiment, to try something, claim it succeeded, and then 
insist --loudly and by whispering campaign-- it was a success 
and that IESG failure to immediately adopt it indicates that all 
of those rascals should be thrown out and the procedural 
structure blown up (and replaced with one of the demagogue's 
liking) to mark their way.


--On Friday, 06 June, 2003 16:15 -0700 Dave Crocker 
<dhc at> wrote:

> Melinda,
> MS> I'm not crazy about things like what's going on with the
> MS> SIRS proposal (going off and collecting volunteers without
> MS> real agreement from the cohort that this is a good thing to
> MS> do) because it feels pre-emptive and unco-operative,
> Oh boy, do we have different views about such efforts.
> Let's be clear about the nature of this sort of "preemptive"
> and "uncooperative" effort:
> 1. At the last IETF meeting, Brian came up with an idea,
> chatted it up, got good reactions and suggestions, recruited
> someone to help with the writing a few paragraphs on it and
> then circulated the idea informally.
> 2. The writeup was chatted up, got good suggestions and
> reactions, and turned into an I-D.
> 3. The I-D got chatted up, got good suggestions and reactions,
> and turned into an experiment.
> 4. The experiment has, so far, gotten volunteers, started to
> offer itself for public use, and will prove useful -- or not
> -- on its merits. It has no force of law, or anything other
> than basic credibility.
> It also has not had months and months of delay.
> So, in fact, this is about as far as you can get from being
> preemptive or uncooperative.
> On the other hand, yes, it is also very far from relying on
> bureaucratic approval.
> Rather than being worried that it has occurred, perhaps we
> should be worried that more such efforts have not?
> d/
> --
>  Dave Crocker <mailto:dcrocker at>
>  Brandenburg InternetWorking <>
>  Sunnyvale, CA  USA <tel:+1.408.246.8253>,
> <fax:+1.866.358.5301>

More information about the Problem-statement mailing list