Staying on Track (Re: Documenting pilots (RE: pausable explanation for the Document Series))

[John C Klensin]

--On Saturday, 07 June, 2003 16:19 +0200 Harald Tveit Alvestrand 
<harald at alvestrand.no> wrote:

>> As I hope my recent notes make clear, I'm not in favor of
>> trying to invent procedures to alleviate any of the above, or
>> of delaying things until such procedures reach some
>> extraordinary consensus.  But I'd be much happier about the
>> idea --even as an experiment-- had the IESG been willing to
>> stand up and say "fascinating idea, lets try it".
>
> Full disclosure: I said "fascinating idea, let's try it". Or
> the semantic equivalent thereof. I thought at the time, and
> still think, that the IESG doesn't have to give blessing to
> this experiment - so I spoke only as myself.

Good to know.  I agree that the IESG shouldn't have to bless 
these things.  But it _might_ have been helpful had your 
observation/ comment been a bit better advertised.  For many of 
these things, I (and perhaps others) get nervous if there is no 
appearance of _anyone_ on the "inside" saying, at least, "ok, 
doesn't seem harmful, try it".   If at least someone does, and 
no one speaks up and says "terrible, risky, doom, 
destruction,...", then going ahead seems rational.   Put 
differently, I don't see any need for formal IESG consensus. 
But an informal sanity check with the IESG seems prudent.

>> And it
>> would have been even better, at least from an "experimental"
>> context, had the IESG chosen to say "It would be good to
>> really do this as an experiment and therefore to be able to
>> make some comparisons. So let's look at the following mix of
>> areas, or specific groups of WGs, first, leave the others
>> with status quo, and see how things work out.
>
> Given the variability between areas that Eric Rescorla's
> numbers showed up, and the variability between groups, I think
> this is difficult to do. I'd be very interested in seeing
> someone propose metrics to measure whether or not the
> experiment succeeded. But I don't really see why the IESG
> should be the one doing that.

I didn't want to suggest that the IESG should do any 
evaluations.  But, if the IESG has insights about why some 
things take longer than others, and what might or might not 
impact that, which have not be shared with the community (and my 
impression is that few of them have), then it is probably time. 
With comments like the above, please distinguish between things 
that I think it would be good if the IESG would do if they knew 
how and had the time and things that I think the community 
should task the IESG to do, feasible or not.    If I wasn't 
clear enough about the above falling into the first category, I 
apologize.   And "that type" below referred to some response 
along the lines of "go ahead and try it", not the experimental 
design.

>> I'm disappointed that type of response from the IESG hasn't
>> happened. Maybe they don't think they have been asked.   And
>> the latter is, of course, what my recent notes have been
>> about: if we need a formal document and request to publish it
>> as an RFC and/or a formal WG call for consensus and response,
>> in order to get an IESG response to a "we think this would be
>> worth trying, do you have reactions or
>> suggestions/requirements about how to try it", then, IMO, we
>> are in deep trouble.    But, going ahead with something on
>> the grounds that it is an experiment, without objective
>> evaluation criteria or any real possibility of them, and
>> without IESG comment, input, or participation...  That seems
>> to me to be an opportunity for some future demagogue, with
>> some other experiment, to try something, claim it succeeded,
>> and then insist --loudly and by whispering campaign-- it was
>> a success and that IESG failure to immediately adopt it
>> indicates that all of those rascals should be thrown out and
>> the procedural structure blown up (and replaced with one of
>> the demagogue's liking) to mark their way.
>
> Point.
>
> *Someone* needs to do quality control and measurement on
> experiments. But .... the IESG has other jobs. Our
> standard-and-often-recommended practice when asked to do
> something is to figure out who to ask to do it.

I wish that "figure out who to ask to do it" were true; I think 
significant evidence contradicts that assertion.  But that is a 
different problem and thread.

> Who should be asked in this case, and who should do the asking?

Harald, my notion of "propose an experiment" is that the people 
doing the proposing are obligated to propose the mechanisms by 
which it can be evaluated and its success or failure determined. 
Without that, it isn't a proposal for an experiment, it is a 
suggestion about some sort of more-or-less ritualized exercise. 
The SIRS draft is especially interesting in that regard because 
it makes a series of assertions in likelihood terms.  Are those 
assertions plausible?  Yes.  But likelihood assertions need to 
be testable -- at least going out and maybe coming in.  And the 
responsibility for demonstrating how they can be tested lies 
with the authors, not the IESG.   The IESG's sole responsibiity 
in this area, IMO, is that, if one of these things comes along 
that the IESG needs to process (and that doesn't included 
informal questions, or individual submissions to the RFC 
Editor-- although I would hope that they would apply similar 
criteria), and it proposes an "experiment" without a clue about 
evaluation, that ought to be sufficient grounds to bounce it.

All of that is pretty theoretical.   What I'm worried about 
specifically is the risk of bad experimental chains tying our 
ability to make rational protocol or procedural decisions in 
knots.  There, SIRS is a lousy example: it seems like a good 
idea, and the authors have been pretty careful about review. 
If someone can say "we will go off and try this 'experiment'"... 
"ok, it succeeded, therefore I invoke the 'running code' 
principle and the IETF needs to adopt it", we slide toward a 
process model that I find almost as frightening as I do the 
canonizing of any statement the IESG, or any AD, or the 
Secretariat, chooses to make about procedures or the belief that 
we can't do anything without formal community action.  We can 
tie ourselves in terrible knots at either extreme.

     john