WG last call - problem description document

[Keith Moore]

1. The document title is misleading; it should be changed to something
like "Perceived Problems in IETF" to make clear that while these
perceptions are widely held, there's not consensus on the list of
problems.

2. Remove the text

   The result will be used to guide
   the next stage of the process in the Problem Statement working group
   which is to recommend the structures and processes that will carry
   out the corrections.

from the abstract - as it is presumptive.

3. In section 2.2, the following text:

   o  The charter may not be sufficiently detailed to document the
      process and timeline to be followed by the WG.  Additional
      documents may be needed such as a roadmap or detailed plans.

could be read as indicating a problem with WG charters.  It's more 
realistic to say that there's a natural gap between the level of detail
specified in a WG's charter and the level of detail specified in a WG's
first draft protocol specifications, and our WG processes do not
currently give much guidance as to how to narrow that gap.  

IMHO IETF's biggest problem is that its working groups rarely develop
specifications by any kind of incremental refinement.  Proposals are
expected to be fairly complete when they first appear as
Internet-Drafts; otherwise they are shot down for lack of completeness. 
On the other  hand, detailed proposals are less likely to get sufficient
review because they are lengthy, and sometimes we shoot down entire
proposals  because they get some of the details wrong - because those
details were specified prematurely.  (And given that few people can be
found to read detailed proposals at early stages, is it any wonder why
we consume precious meeting time with boring, detailed presentations?)

Working groups should be expected to take problem descriptions at the
charter level and iterate them through increasing levels of detail, e.g.

- problem definition/statement
- goals document (NOT requirements, though there may be a few goals 
  that are non-negotiatble)
- functional specification / block diagram
  that broadly specifies how the functional pieces interact 
- rough specification
  which might specify protocol elements and their semantics but not to
  exact detail of presentation
- prototype specification
  which specifies a subset of the protocol in enough detail that it
  can be implemented and tested (without being ready for prime time
  in any sense - for instance, things like authentication might be 
  specified in a trivial fashion at this phase, even though security
  goals and requirements need to have already been specified) 
- draft specification
  which is a fairly complete specification for the protocol as it
  is intended to be deployed but may have bugs or a few unspecified
  areas
- PS candidate specification
  which is believed to meet the requirements for Proposed Standard

This seems like a lot of documents, but the first several of these 
documents should be quite short.

Cross-area review should happen early, probably by the functional
specification phase and certainly by the rough specification phase.

Early documents are not carved in stone, they should be revised
as it becomes apparent that changes are needed - so that there is
always a current problem definition, goals statement, functional 
spec, etc.

Not every WG should have to do each of these - it depends on various
factors such as the overall complexity of the group's work, the 
size of its user community, the amount of interaction with other
protocols and other areas, whether there are compatibility issues,
etc.  The important thing is the pattern of incremental refinement
and review, not the specific set of steps.

suggested action: reword to something like:

   o  Currently there is no expectation that WGs define their
      processes or timelines beyond the level of detail specified 
      in the group's charter; however this level of detail is rarely
      sufficient to direct a WG's activities.   Additional documents
      such as a roadmap or detailed plans are rarely produced.

4.  same section

   o  Charters regularly fail to set sufficiently granular milestones at
      which progress of WGs, individuals and documents can be evaluated.
      Also WGs often do not make more detailed work plans to refine the
      charter plans.

Again, while true, this seems phrased to put the burden on charters and
I don't think it belongs there.

I don't think it's reasonable to specify this level of detail in the
WG's  charter. To be useful, dated milestones need to be expressed in
terms of  meaningful, measurable results, and "submit an Internet-Draft
on topic XXX", while measureable, isn't meaningful.  To be meaningful
the goal needs to be understood well enough in terms of things that have
been done before.

IMHO the charter should specify an expectation of the minimum steps that
a WG will follow in producing a specification (say, in terms of the list
above) and a vague expectation of the timeframe in which the steps will
be undertaken - but should not be expected to serve as a detailed 
schedule.  Instead, WG chairs should amplify on the vague outline 
provided by the charter and produce schedules (separate from the charter
outline) for the next few months outlining specific, measurable steps 
(revise functional spec; decide on issues X, Y, Z; review documents
A, B, C) and timeframes that actually have some basis in reality.

suggested action: reword to something like:

   o  The timeframes specified in charter milestones are generally not
      grounded in reality, nor is it reasonable to expect tight
      adherence to schedules that are specified at the level of detail
      appropriate for a working group charter.

5. section 2.4

   o  The three stage hierarchy is, accordingly, seen to be excessive.

we really haven't demonstrated that three stages are excessive;
what we've seen is that the stages we have are a poor impedance match
between our energies to refine our output and the input accepted by
our "customers"

suggest change to:

   o  The three stage hierarchy thus appears to be a poor match for
      vendor expectations, market expectations, and/or energy to
      complete the process.