Is this really where we want to go? (was: Re: Selecting leadership, take 2)

[Harald Tveit Alvestrand]

Thanks John - thoughtful thoughts should always be welcome.

--On 6. desember 2002 21:38 -0500 John C Klensin <john-ietf@jck.com> wrote:

> Harald,
>
> Sorry this is so late in the game, but I've been thinking about where
> this "let's form a WG to do problem definition" approach seems to be
> taking us in the context of a number of off-list interactions, which were
> reinforced by part of Marshall's comments earlier today.
>
> Summary for those trying to figure out whether to read this note: I have
> concluded that the whole idea of creating a problem definition WG is
> wrong: probably a waste of time and certainly a distraction from the way
> in which we should be focusing our efforts.  I am going to try to explain
> why, then propose an alternative.
>
> I suggest that there are some things the IETF historically does rather
> poorly.  This list isn't of problems we need to solve, but of things we
> need to recognize and work around, lest they swamp us.  Certainly it is
> unlikely that any change in either actors or processes will make us
> better at these things (although some might make us better at declining
> to try to do them).  For example:
>
> 	* We have discovered that Working Groups are rarely a
> 	good way to get from <interesting-buzzword> to a serious
> 	set of definitions and requirements.  Working groups
> 	charged with coming up with requirements in ill-defined
> 	areas often go on rathole tours and even more often take
> 	much longer than anyone expects to produce anything.
> 	And the results are rarely suitable for real evaluation
> 	and examination, as evidenced by the number of
> 	subsequent "protocol" WGs who end up ignoring or
> 	rejecting a significant fraction of the "requirements".

Agreed. My personal formulation is that "any WG, no matter how large, 
consists of ten people" - for any working group that WORKS, you will find a 
much smaller team that does the heavy lifting to produce something coherent 
and rational, with the WG making an important contribution to:
- make sure important things get mentioned
- make sure important things, once mentioned, are not forgotten
- make sure the (lowercase) I's are dotted and the t's are crossed

> 	* We have discovered that Working Groups are rarely good
> 	at doing real "blank sheet" design.  Either some design
> 	shows up from outside the WG and the WG endorses it,
> 	perhaps with some tuning, or the WG ends up going around
> 	in circles until it either produces garbage (which gets
> 	approved because everyone but a few advocates are
> 	exhausted and no one has the heart to say "no" because
> 	so much time and energy has been invested.

Agreed.

> 	* And we have discovered that we don't do "thinking
> 	about process" WGs very well.  Given a clear problem and
> 	a forced schedule, such as was the case with the
> 	original Poised group, we can often figure out something
> 	decent.  But, with a looser schedule and a vague problem
> 	definition, we wallow, nit-pick, practice becoming
> 	amateur lawyers, and spend huge amounts of time on
> 	things that are unlikely to make very much difference.
> 	And we often spend even more time looking at cases that
> 	haven't occurred and are unlikely to do so.  The more we
> 	wander in those weeds, the more we devise "solutions"
> 	that aren't testable -- we may have code, but we don't
> 	have a clue as to whether or not it actually runs.  The
> 	last couple of years of Poisson were a fairly good
> 	example, but we have many more specific ones.

Agreed. Which is one reason why I deliberately did *not* propose that the 
*next* step in the process (the one beyond figuring out what the problem 
is) be a WG. I think we should have that debate.

> I'm not trying to start a discussion here about whether we should avoid
> attempts that fall into those categories at all, or how they should be
> constrained and managed.  I just want to see if we can get agreement that
> these are areas in which we don't have a good success record... indeed,
> that we have a significant and regular record of failures.
>
> So, what are we proposing to do?
>
> We are going to create a WG whose purpose is to figure out what the
> problem is, so that we can create a WG to design a solution and then,
> either in that WG or in some other one, we can use a WG to start writing
> procedures.  Two things are wrong with this: each of the steps falls
> squarely into our areas of incompetence as identified above and, given
> the tendency of things in those areas to drag out, the solutions are
> likely to be outdated by the time we discover them.

Well, I think the WG paradigm is fairly good at getting the right words 
shouted (together with about 10 times as many wrong ones). But the WG 
paradigm alone is only part of the puzzle - which is why I keep harping 
about figuring out what leadership we want for this task; the person who 
takes charge will have the task of finding the right people to do the 
filtering, winnowing and structuring job - not easy.

> If we think there is a problem, or some set of problems, that are
> important enough that we would like to see them dealt with soon  (e.g.,
> certainly within the next calendar year and preferably before a new IESG
> is seated), that strategy borders on the silly.  I know it is not
> Harald's intent, but the effect would almost certainly be to postpone any
> real improvements -- even the obvious ones -- on the grounds that the WG
> is thinking about the grand issues.  Good way to do philosophizing;
> terrible way to do engineering.  Philosophizing isn't supposed to be our
> goal around here.
>
> Radical suggestion:
>
> Forget the "big problem", "big picture", and general surveys of what is
> wrong with the IETF and stop feeding whatever lives at the bottom of the
> rathole.  Assume that the current general structure/ framework is good
> enough and that what we need to do, at most, is some tuning, not a "blow
> it up and start over" approach. (Skeptics should remember that, before
> the Kobe affair and the Boston Tea Party, we had an IAB, an IESG made up
> of Area Directors, and Working Groups.  And, when we got finished, we had
> an IAB, an IESG made up of Area Directors, and Working Groups.  Yes, a
> lot of roles changed, and things got retuned, but no large changes in the
> structure on which things are hung.)
>
> Instead, note that the PACT document, my response to Geoff and Marshall,
> Mark's note, and several less comprehensive (in the sense of shorter and
> with fewer individual points) postings can be decomposed into a series of
> statement sets of the form of:
>
> 	* X appears to me/us to be a problem
> 	
> 	* Because of X, the following bad things happen: A, B,
> 	C,...
> 	
> 	* One could do Y, which would change or modify X so that
> 	those things would not happen
>
> Now, it seems to me that, whether we do it as a WG or simply on this
> list, we ought to be able to look at each of those sets in turn (and any
> others that can be proposed).  Stated that way, serious discussions about
> each element are possible: Is X really enough of a problem to bother
> with?  Can we agree that A, B, and C are actually consequences of X, or
> might they result from something else that needs to be looked at?  Does Y
> really solve X, or does it just attack a subset of A, B, and C?  And are
> there side-effects to Y that are worse than X itself?  Is it safe to
> deploy Y by itself, or are there interactions with other things that
> should be considered first (and what are they)?

If you remove Y from the equations above, don't we have a fairly good start 
at a problem statement?

My worry is the set of Y that translates to "if you work a little bit 
harder, all will be well" - for instance the "AD should inject 
architectural clue earlier" suggestions, while in fact very reasonable in 
themelves, are in this category. Like any good budgeting process, they need 
to be paired with "work a little less hard on Y'" proposals.

> That is a case-by-case analysis problem with well-understood types of
> questions.  It is fairly close to what we try to do with network
> protocols being proposed for standardization.  If we care, we can usually
> deal with those sorts of problems fairly effectively.  And it is very
> different from going off into the weeds hoping that we can find a crisp
> problem definition out there somewhere and do it in finite time.
>
> So I would suggest that:
>
> (1) We either start that process here, or spin up a WG whose function it
> is to start working through the proposals.
>
> (2) We adopt approximately the same rule toward proposals to be
> considered that we usually adopt in well-managed WGs, i.e., nothing gets
> discussed unless there is a clear written statement of it, and, in this
> case, that statement contains the "problem - explanation of consequences
> - proposed solution and its implications" statements implied above.
>
> (3) That process, and the proposals/statements, try to clearly
> distinguish among
>
> 	* issues that can be addressed by different attitudes
> 	among, e.g., IESG members,
> 	
> 	* issues that can be addressed by additional guidance to
> 	the IESG, and
> 	
> 	* issues that require changes in how we are organized
> 	and the ways in which we do things.
>
> When we can reach agreement and there don't seem to be complex
> interactions, the first category should be treated by the Nomcom, and the
> second by the IESG, as real-time input with the expectation that they
> will react in real time.  The third obviously will take somewhat longer,
> but perhaps we can start seeing the beginnings of changes and results by
> the time we see each other in San Francisco, rather than getting there
> with just the hope of a report at a plenary to be followed by more
> discussion and then by another WG.

one word on fixes:

I don't want to stop improving things while this goes on.
We'll always be working with things like tracking improvements, chasing old 
stuff down, resolving comments speedily, adjusting attitutes, working on 
training and so on.

as you say, there are lots of problems that do NOT need any changes to the 
process to solve.

but I don't think those are the underlying problems that led us to where we 
are now. And I'd like to have those talked about.

               Harald