General comment on draft-ietf-problem-statement-00.txt

Keith Moore moore at cs.utk.edu
Sat Mar 1 16:01:31 CET 2003


> >- due to IESG workload and breadth of IETF's scope, most documents are only
> >reviewed in detail by one or two IESG members anyway, so objections from
> >one or two members necessiarly carry more weight than it would otherwise
> >seem.
> 
> Well, this is a straightforward statement of the workload problem. 
> You're saying that the IESG is so overworked that finding a problem 
> in a document is rather lucky: Only a couple of IESG members will be 
> reviewing it anyway, so if they happen to be the ones that have the 
> expertise to find problems, we ought be glad they caught it.

This is not quite what I was saying.  IMHO, having one or two people review a
document in depth, as a back-up for thorough WG and last-call review, is
more-or-less sufficient.  At the same time, when I was on IESG it was not
unusual to get a document which clearly had not benefited from sufficient
review at any level - either in the WG or in last call - and in this case the
ADs reviewing the document bore almost the sole burden of doing review and
recommending changes to make the document adequate for IETF blessing.   And
that's clearly too much burden to place on one or two people.  

But it would be a huge stretch to infer from this the problem could be solved
by diluting IESG authority. 

> Workload then seems like something that needs to be addressed.

Workload is definitely something that needs to be addressed - at all levels.
We need to have realistic assessments of the amount and kinds of work that
we're willing/able to do.  We might need to figure out how to attract more
people, or people who are interested in doing work that isn't being done well
now.

> >It's not unusual for an AD to say "I trust the ADs in that area to do due
> >diligence in fixing that document".
> 
> Why is it then that AD's are not inclined to say "I trust the chair 
> of that working group to do due diligence in fix that document"? 

They do.  But WG chairs tend to be narrowly focused.  In general the chairs do
due diligence to make sure that their drafts satisfy the working group, and
are technically sound within that narrow focus, and in general ADs trust them
to do that.  But that's not sufficient.  For instance, when I was on IESG it
was not unusual for a WG to completely fail to address security issues, or to
fail to consider the impact of a protocol on other concerns  (e.g. using
HTTP as a means to bypass firewalls).  ADs have a broader perspective, and
when reviewing a document they are more likely to look for things that impinge
on other groups, other areas, or other interests.  It's possible to get this
kind of review/feedback in Last Call also, but it happens too rarely.

> Perhaps this indicates another problem, that we do not have 
> adequately qualified WG chairs that ADs can trust to do the real 
> review (cf. 2.6)? This does seem like an area where "too much 
> concentrated authority" (lack of delegation) is part of the problem. 
> (2.4)

As far as I can tell in our current setup WG chairs suffer from fairly
significant, and inherent, conflict-of-interest problems.  They need to be
motiviated to work on the problem (and often have strong ideas about what the
outcome should be) but are expected to not bias the group toward some
particular approach or another.  They tend to be chosen by the (often
ambitious) working group but are expected to take direction from an AD whose
concern may be to keep the group within its scope and to keep it from
interfering from other groups.  They are expected to be advocates for the
group's work to IESG but are also expected to implement changes requested by
IESG.   Good chairs are *very* hard to find.

Concentration of authority is only a small part of this problem.  Most WGs now
have multiple chairs, partially to reduce workload on a sole chair, partially
to
keep one chair from steering the group in his/her own direction.  IIRC all
areas in IESG now have multiple ADs, partially to reduce their workload,
partially to keep one AD from be the sole dictator of policy in his/her area. 
Both of these measures have IMHO helped.  But adding more chairs makes a WG
unwieldy, and adding more ADs makes IESG unwieldy.  I'm fairly convinced that
IESG is too large, or at least, as large as it should get.  There are so many
people in IESG that it's difficult to have a useful discussion, so many that
it's difficult to get together for a meeting, so many that the potential for
ratholing has become significant.  And yet, I think that for other reasons
(like needing to be comprehensive and have multiple ADs per area) it needs to
be about as big as it is.  Non-essential areas can be pruned, but all of the
current areas seem to be essential.

> >- Sometimes an AD objects to something in a document because he doesn't
> >understand some subtle aspect of the design or the compromises that were
> >deemed necessary by the WG.
> 
> This may be the fault of a document editor not appropriately 
> documenting a compromise choice so that AD can figure out why the 
> choice was necessary. That may amount to a 2.5 kind of problem.

When I was on IESG (this may have changed) there was a serious lack of
communication between document editors and IESG.  The AD for an area, rather
than the WG chairs or document areas, was expected to advocate the document to
IESG.  This is really difficult if the AD has reservations about the document.
It also imposed distance and bandwidth limitations between the AD who was
complaining about the document and the people who were in a position to fix
the problem or explain the reason for the document being the way it was. 
Sometimes there would be a conference call set up between the ADs who cared
and the WG chairs and the document editors, but this was difficult to arrange.
Everybody is busy.

> >Given workload and scheduling constraints (and
> >sometimes stubbornness) it can (quite unfortunately) take months to work
> >these things out.
> 
> Though I understand it was parenthetical, I am quite concerned that 
> "stubbornness" was a feature of the process that you thought worth 
> mentioning. I understand that ADs are human beings and have the same 
> flaws as the rest of us.

ADs aren't the only ones who suffer from stubbornness.  There are at least as
large a proportion of stubborn WG chairs and document authors as stubborn ADs.
 Also, stubbornness is sometimes a virtue, sometimes a vice.

> However it seems absolutely essential that 
> we choose ADs who can get by certain human failings and say, "The AD 
> who put this forward is not stupid, yet I still think there is 
> something flagrantly wrong here. Perhaps I should go talk to the 
> chair or the document editor to see what's going on before I assume 
> that I'm the only sane one on the IESG."

This does happen.  Most of the IESG people I worked with were capable of doing
this, much of the time.  But we all have our blind spots.

>  I am worried that we are not recalling ADs who are too 
> "stubborn". They are an impediment to good quality work being done in 
> a timely fashion.

The same could be said about stubborn WGs and stubborn WG chairs.

> >- Sometimes one AD attempts to slip something past other ADs but only
> >one or two ADs notice and are willing to invest the time/energy to object.
> 
> OK, so my chin hit the keyboard when I read this. Surely you're 
> kidding. If we've got ADs trying to slip something past other ADs, 
> those folks should be instantly recalled. 

There's no way to prove that this is going on, but occasionally I suspected
it.  Maybe the AD who is trying to slip something past thinks that he's
doing the right thing by trying to keep it from being noticed by the AD who is
likely to object to it.  Or maybe he has ulterior motives.  I don't think it
happened very often; then again I was occasionally accused of being naive.  

I don't know any way to fix this problem structurally, at least that hasn't
already been done.  Two ADs per area helps.  Allowing any IESG member to vote
Discuss on any document from any area also helps - it means that there is a
significant risk of push back when such things are tried.  But making it
easier to recall ADs just seems likely to make the politics nastier - it would
be another tool that could be used by those who play hardball, and it would
probably get rid of more good ADs than bad ones.  There's something to be said
for stability.

I respected every AD I worked with - even those who often seemed to be getting
in the way.  Some of them were pains in the wazoo.  Every one of them was
valuable, and as far as I could tell the level of competence was high.  I
don't think it's possible for IESG to not be political.  I was generally
impressed by the degree to which the discussion focused on technical issues
even though I was too often frustrated by the occasional heavy-handed
politics.  And I was probably as heavy-handed as anyone else.  I did what
seemed to me to be best for the Internet, and I assumed that most of the time,
most of the others were doing what they thought best.   I burned myself out
doing it, and so did some of the others.

> Now, maybe you're saying they have to do this 
> because of incorrect behavior on the part of the other ADs, like 
> objecting to documents without justification, but in that case those 
> bad behaving ADs ought to be removed. 

There are no objective criteria for what is good and bad here.

> Short of that, an AD who is so 
> irresponsible that they try to slip something by on the others should 
> be out on their rear immediately.

And again, there are no objective criteria for this.  The AD can always claim
to have simply missed it himself, or to think it's not a problem.  And the
only people who are in a good position to notice are IESG people.  The last
thing we need is to create more conflict within IESG.

> In addition, it seems like if one or two ADs *do* notice such an 
> occurrence, it should be  easy for those one or two to convince the 
> rest of the ADs what's going on. In that case, we shouldn't be seeing 
> 1 or 2 discuss votes, we should be seeing 12, shouldn't we?

Nope.  First of all, you will rarely get unaminity.  Second, voting discuss
means more work, so you're not likely to vote discuss if you think someone
else is on the problem and will do due diligence to get it fixed.  Third,
because every discuss vote is a potential barrier to getting a document
approved, ADs with objections will often vote "no objection" and explicitly
say that they trust some other AD who has voted discuss to fix the problem -
purely to streamline handling of the document and get it out the door as soon
as it is fixed - to not hold it up for another two-week cycle, to not take up
valuable telechat time polling all of the ADs who have discuss votes, so that
the time can be used for discussing other documents.  (you *do* want this)

> Did I miss something in what you meant?
> 
> >- the nature of the IESG decision-making process is that ADs who object to
> >a document are expected to explain what it will take to fix the document.
> >sometimes (too often!) the protocol is so flawed, or the document so poorly
> >written, that it simply can't be fixed by making a few tweaks.
> 
> Those seem like 2.2 problems.

Aside here - I'm not looking at the document to see whether these sections are
appropriate but I'm not convinced that the current document structure helps to
illuminate the nature of the problems.   If I can find a better structure
I'll suggest it, or I'll write a draft of my own.  Meanwhile, if you're trying
to argue that the current structure is sufficient, well, I'm going to reserve
judgement on that.

> >so the ADs who
> >want to be conscientious (and not simply approve the document) end up
> >making vague or seemingly trivial objections, because there isn't time
> >enough for the AD to rewrite the document.  (and the WG doesn't know what
> >to do with these objections).  other ADs just hold their noses and vote 'no
> >objection' because providing useful review and feedback given the
> >constraints of the balloting process is either infeasible or too
> >time-consuming.
> 
> So to address (compound?) the 2.2 failures, most ADs hold their noses 
> and let the document go through (that's pretty irresponsible) and one 
> or two ADs vote discuss, but don't make reasonable objections to the 
> document. That's horrible.

You may think it's horrible or irresponsible,  and I have immense respect for
you and your opinion.  But if you haven't been there,  you are not likely to
understand why it's the way it is.  Every AD knows (or should know) that his
own judgement is limited, that sometimes his objections are misplaced, and
that there's a limit to his  ability to anticipate problems (or lack of
problems) with a protocol before it is deployed.  Sometimes "holding your
nose" is just this kind of admission - it's a way of recognizing that your
judgement has limitations.  This seems entirely appropriate, and if ADs did
not sometimes hold their noses, document delays would be much longer. 

Then again, there's also the consideration that every time you object to a
document you are making more work for yourself and your fellow ADs, who will
not appreciate it, and who may either retaliate by making your life harder, or
may simply trust you less (which amounts to almost the same thing).  So you
tend to let some things slip by even if they make you uncomfortable. In some
ways this is a good thing - it means that ADs have incentives to use their
discretionary power sparingly.  But sometimes it may also mean that
documents don't get adequate review.  Balance is *hard*.

> With this, and your previous comment, perhaps these two results (the 
> IESG response) of the 2.2 problem need to be described fully in 
> appendix A. However, I do think the document gets at the root cause 
> of the problem and identifies what we need to address when we get to 
> the solution phase.

I think the document misses so much of what is going on, that in its current
form it is not useful input toward a solution - actually I think it's likely
to produce solutions which do more harm than good.  It's a good start
precisely because its omissions highlight the need for more research, and
because it will stimulate more discussion.  But another iteration or two is
definitely needed.  

In particular, I think people need to understand how IESG works - even the
dirty side of it - in order to have an accurate picture.  But focusing too
much on IESG will miss the point, like trying to understand the elephant by
feeling its trunk.  We need to see the whole elephant, including its nether
parts, but we should not focus on the nether parts.  IESG does not exist in a
vacuum.  Many of the problems seem to be at the interface between IESG and
WGs/authors/chairs.  Many of them seem to be that documents don't get adequate
review either within WGs or within IETF as a whole before being sent to IESG. 
Many of them seem to be that WGs don't stay within their charters.  Many of
them seem to be with the way discussions are conducted within working groups. 
Fix these problems, and document quality will go up, IESG's workload will drop
considerably, IESG will be placed in difficult positions much less often, and
IESG people will probably get more patient and have more bandwidth to work out
problems when they do occur.  But you can't fix IETF without looking at the
whole picture in more detail than is presented in the current document.

(And I'm not saying that structure changes should be ruled out.  But I think
we're still going to need an IESG, which has a very similar makeup, and
similar responsibility, to the current IESG. )

Keith



More information about the Problem-statement mailing list