More detailed comments on the Problem Statement document

[Charlie Perkins]

Hello again folks,

In general, the document is quite verbose.  A spell checker should
be employed, and ways should be found to eliminate redundancy.

Here are some more detailed comments on the draft.

In section 2.1, I do not like the following sentence:

>    o  The misty vision has inhibited the development of roadmaps that
>       would inform the IETF's stakeholders of our longer term
>       intentions,

In the first place, I don't think the IETF has many longer term intentions,
except perhaps "saving the Internet".  Such intentions don't necessarily
need "roadmaps".  I think the IETF is much more reactive than would
be understood from the above sentence, and I think it's a good thing for
the IETF to be that way.  This is not to say that the IETF can only solve
problems after they occur.  I do think think that the IETF should try to
prevent problems, as far in advance as we reasonably can.  But the
IETF actions are happening in response to identified problems, not
(in general!) according to some over-arching roadmap, architecture,
plan, or vision.

In section 2.2, on page six, there is the following text:

>    o  Failure to identify and articulate engineering trade-offs that may
>       be needed to meet the deadlines that the WG has set without
>       inappropriately reducing the 'fitness for purpose' for the
>       intended customers.
>
>    o  Continued refinement of the solution beyond the point at which it
>       is adequate to meet the requirements placed on it by the intended
>       purpose.


I am concerned that listing these as bullets needing attention is
counterproductive.  For one thing, most engineers are aware that
these problems need to be avoided.  On the general assumption that
most WG staff (i.e., the ones that do the work and that care) are
engineers, there isn't much need to restate the obvious, and most
people will not derive much useful content from these bullets -- i.e.,
it's the negative equivalent of "motherhood and apple pie".  I don't
remember when anyone consciously acted in the above bad ways.

In the next list:

>    o  Poorly defined success criteria for WGs and individual documents.

I'm not sure this is really worth calling out as a specific problem.
For one thing, the success criteria are the charter items.  For another
thing, if there is a poorly defined criterion, it stems more from the
problem with the standards track than anything else, and that's
not going to get fixed by rewhacking WG processes.

To the point:

>    o  Lack of review, especially early review, by reviewers who are not
>       directly interested mebers of the WG, and by subject matter
>       experts for topics related to, but not necessarily the immediate
>       focus of the document.

I would like to suggest that we not view this as a "problem", because there
might not be a good solution.  Good external reviewers don't grow on trees,
and if they did anyway we would find the trees are already completely
plucked for other reasons.  I don't see how we can expect to get so many
good external reviewers that they could spend time reading early drafts that
aren't even implemented by WG members yet.  I'd much rather see such a
valuable resource be put to use after some initial validation within the 
working
group itself.

Another way to attack this deficiency, as I have tried to suggest before, is
that as time goes on, we amass a body of documentation by subject matter
experts that explain how to avoid certain cross-working-group problems.
It's a lot more scalable to ask engineers to try to find relevant documents
that have been written about a relevant area of cross-expertise 
interactions,
than it is for them to find an expert to explain it for the millionth 
time on the
expert's personal timeline.

So, I'd replace that bullet by something like:

o  Lack of documentation about likely problem areas that might arise
    due to interactions with other popular IETF protocols.

The point, on page 12

>    o  Absence of documented debriefing sessions to assess and record the
>       successes and failures of WGs (and other processes) so that the
>       positive and negative lessons learned can be used to facilitate
>       future work and improve the processes.

is essentially the same as the previous point, and both could be expressed
more succinctly as:

o  Lack of a WG "post mortem" procedure to drive the improvement of
    the standards development process.

The following point:

>    o  Lack of criteria for determining when a piece of work is
>       overrunning and/or is unlikely to be concluded successfully either
>       at all or within an acceptable time frame: Lack of process for
>       either extending the time frame, adjusting the scope or
>       terminating the work item or the whole Working Group.

is debatable, because the process should be "amending the charter",
and the criteria should be "according to the charter".

On a positive note, I wholeheartedly agree with the following point,
and I suggest that current efforts towards improvement be encouraged
and extended:

>    o  Automated tools to support the engineering process are minimal.

Regarding the next bullet item, I would suggest that no change is needed
to current IETF processes:

>    o  Despite its commitment to 'running code', the IETF is not
>       proactive in providing ways for developers to verify their
>       implementations of IETF standards.

So far as I can tell, the developers are quite adept at making this
happen regardless of whether the IETF helps or not.  Or, as they
say, "If it's not broke, don't fix it".

Also on page 12, the following bullets are redundant and could beneficially
be deleted to avoid numbing repetition:

>    o  Project entry, goal setting, dependency identification,
>       coordination and tracking processes are all either missing or
>       implemented less effectively than the norm for commercial
>       organizations in related activities. Dependencies and coordination
>       should cover both other WGs within the IETF and any outside SDO
>       with which the IETF is collaborating.
>
>    o  Charters regularly fail to set sufficiently granular milestones at
>       which progress of WGs, individuals and documents can be evaluated.
>       Also WGs often do not make more detailed work plans to refine the
>       charter plans.
>
>    o  The acceptable deadlines for finishing a piece of work, and the
>       criteria used to determine them, are rarely, if ever, documented.
>       Also the estimates of the time required to complete the work often
>               .......

Unless I am missing something, all of these ideas have already appeared
earlier in the section.

On the next page, I disagree with the following statement:

>    One problem which the IETF does not appear to suffer from is
>    excessive bureaucracy, in the sense that transfer of information is
>    generally kept to the minimum necessary to accomplish the task. 

For one thing, I might suggest that the current "rfc-editor" processing
is quite bureaucratic, regardless of its value.  Also, the evaluation of
criteria for advancement along the standards track, the submission
process for Internet Drafts, the BOF approval, and other examples
can be viewed as more or less bureaucratic.  Maybe it's not as
bureaucratic as other organizations.  Maybe.

Another problem is that one person's "minimality" may well be another
person's "insufficiency".  Typically, bureaucracy creates procedure based
on historical problem solving episodes.  I can think about the IANA
considerations in exactly this way -- a little bit along the way towards
how mortgage agreements are structured.  Of course, one wants to
avoid ALL systemic problems in permanent transactions.

So, I'd reduce the amount of self-congratulation on this point.  The best
you can say is that the IETF bureaucracy is younger than some of the
other bureaucracies we might be able to name.

In section 2.3, the first paragraph should cross-reference section 2.2
since the topic is related to the lack of external review.  Perhaps better,
the bullet in section 2.2 should be moved here, on the theory that this
is a fact of life, not a problem we can try to solve.

I have another thought about section 2.3 that I have moved into
a separate note.

Regarding the point on pg. 14:

>    o  The IETF does not have an effective means for defining
>       architectures and frameworks that will shape the work of multiple
>       WGs.

... that is something that should be done by the IAB.  So I would say
that the IETF does have the means, but it's perhaps not yet effective.

On page 16, in section 2.5, the following sentence should be deleted:

>    Whilst this might be put down to contributors having less time
>    available in their work schedule during the downturn of the last two
>    years, this is not the whole story because there were signs of this
>    effect back at the height of the boom in 2000.


In section 2.6.4, the last paragraph should be deleted.  It has no place,
and the most charitable interpretation would put it squarely into a 
solutions
document.  More likely, it looks like an insult.

In section 2.7, the claim is made that problems result from the
non-hierarchical nature of the working group.  I don't believe it.
I have never seen this as a problem.  In fact, I would characterize
it as a feature.

If anything, there might be a problem with assignment of tasks,
and accountability for accomplishing those tasks.  Typically,
hierarchy helps with delegation, accountability, and evaluation,
but the lack of hierarchy does not mean the lack of those
three management aids.  In fact, I would in solution space
suggest that the new methods for issue tracking could be tied
in with a non-hierarchical method for delegation, accountability,
and evaluation for better results.

I will send out yet another note with detailed editorial remarks,
with a lot less substance and importance compared to these
comments.

Regards,
Charlie P.