problem statement: Engineering practice vs. management practices

[Keith Moore]

> > 2.2 The IETF does not Consistently use Effective Engineering
> > Practices
> > 
> >    For an organization with 'engineering' in its title and
> >    participants who are likely to trot out the statement "Trust me,
> >    I'm an engineer!" when confronted with the need to find a
> >    solution to a particularly knotty problem, the IETF has, at least
> >    in some cases, extremely ineffective Engineering Practices.
> 
> 
> > 2.7 Working Group Practices can make Issue Closure Difficult
> > 
> 
> I have a hard time understanding the difference between 2.2 and 2.7
> and parts of 2.6. I _think_ section 2.2 is trying to say something
> about how we don't actually use more rigorous and formal internal
> processes that would help us avoid the problem of documents taking too
> long, etc. But one could argue that a lot of that is related to basic
> project management issues (or from Section 2.7 "Working Group
> Practices can make Issue Closure Difficult"), and its not always clear
> whether a cited issue is the lack of a tool (or lack of its use) or
> just lack of use of basic project management techniques, or just bad
> WG practice, for which there might be several ways to try and improve
> the situation.


On rereading these parts of the document I've come to the conclusion
that most of 2.2 is really about management practices rather than
engineering practices.  Nevertheless I strongly agree that IETF does not
consistently use effective engineering practices.  (Maybe the problem is
that many people in IETF are not familiar with the discipline of
engineering or do not know how to apply it to protocol design?)

So I think section 2.2 needs to be rewritten a bit, and some of the 
material in 2.2 needs to be moved to 2.7.  In other words we need
to make sure we understand (and the document is clear about) the 
difference between engineering of technical solution and management
of groups that are chartered to produce technical solutions.  
I think it's worth doing because we really need to promote a better
understanding of the discipline of engineering within IETF.  


IMHO, these are about engineering, though to some degree they're stated
in the language of management more than the language of engineering:

>    o  Failure to ensure that there is a uniform view in the WG of the
>       scope of the WG activity, especially the intended purpose of the
>       solution.   

i.e. failure to define the problem to be solved
> 
>    o  Failure to identify at an early stage (before the design is
>       frozen), and/or then to ensure that there is a uniform view in
>       the WG of the issues that need to be resolved to bring the work
>       to a satisfactory conclusion.
> 
>    o  Failure to identify and articulate engineering trade-offs that
>       may be needed to meet the deadlines that the WG has set without
>       inappropriately reducing the 'fitness for purpose' for the
>       intended customers. 

This is another example where the language is expressed in management
rather than engineering terms.  From an engineering perspective it's
appropriate to think about time-to-specify as one of the many costs of
an engineering solution which influence the compromise that must be
made. IMHO it is rarely appropriate to treat a "deadline" as a concern
that overrrides other costs.  And the cost to the net of living with a
poor solution for a long time should also be considered.

>    o  Continued refinement of the solution beyond the point at which
>       it is adequate to meet the requirements placed on it by the
>       intended purpose.

These are mostly management issues rather than engineering issues:

>    o  The charter may not be sufficiently detailed to document the
>       process and timeline to be followed by the WG.  Additional
>       documents may be needed such as a roadmap or detailed plans.

Yes, WGs need to develop their own roadmaps and plans for the work
they expect to do.

>    o  Poorly defined success criteria for WGs and individual
>       documents.

This is a management issue.  Success criteria for the technical
solutions would be an engineering issue.

>    o  Lack of written guidelines or templates for the content of
>       documents (as opposed to the overall layout) and matching lists
>       of review criteria designed to achieve appropriate quality in
>       output.
> 
>    o  Lack of auditing against explicit criteria throughout the
>       standards development process.

Auditing against technical solution criteria would be an engineering
issue, auditing against other criteria (like document quality) would be
a management issue.

>    o  Lack of metrics to measure the achievement of the desired
>       quality and the performance of WGs and the wider IETF..

Again, measurement of WG performance is a management issue;
measurement of technical solutions is an engineering issue.

>    o  Lack of an explicit feed forward path to drive the improvement
>       of the standards development process and success criteria over
>       time and to avoid repetition of failures that may occur.

Anything having to do with the process is not an engineering issue

>    o  Absence of documented debriefing sessions to assess and record
>       the  successes and failures of WGs (and other processes) so that
>       the positive and negative lessons learned can be used to
>       facilitate future work and improve the processes.

see above.

>    o  Lack of criteria for determining when a piece of work is
>       overrunning and/or is unlikely to be concluded successfully
>       either at all or within an acceptable time frame: Lack of
>       process for either extending the time frame, adjusting the scope
>       or terminating the work item or the whole Working Group.

This conflates process/management and engineering issues.  If a piece of
work doesn't satisfy its engineering criteria (including time-to-produce
criteria) this is an engineering issue.  And it would be appropriate to
make those criteria explicit.  But the lack of process for adjusting
time frame or scope of a group's activities is a process or management
issue.

>    o  Tools to support the engineering process are minimal.

This is very vague - it's hard to know what tools are meant here. 

>    o  Despite its commitment to 'running code', the IETF is not
>       proactive in providing ways for developers to verify their
>       implementations of IETF standards.

FWIW, we've never relied on verification, and this was an explicit
decision. We've placed high faith in interoperability testing, which is
not the same thing.  Of course, we don't do much to facilitate
interoperability testing either.  But we should probably amend this
bullet a bit to say something like "...for developers to test the
interoperability of their implementations."

> 
> >    2.7   Working Group Practices Contribute to Delays
> 
> Some additional examples for section 2.7:
> 
>  - too many WGs have a adopated a cycle where the bulk of the activity
>    is centered around the face-to-face meetings. E.g., Many IDs are
>    revised only once per cycle, immediately before a meeting, with too
>    little active work going on in the months immediately after a
>    meeting.

Saying that WGs have "adopted" this cycle may be giving WG management
more 'credit' than it deserves.   F2F meetings with their imposed I-D
deadlines are naturally going to encourage a tendency toward periodic
releases of drafts, and this is often a good thing.   And for early
phases of operation of most WGs, one draft per cycle can be quite
reasonable.  But when the document is nearing completion it becomes
important to release drafts more quickly, to avoid having to wait many
months to work out small wording changes.  Even more important 
than revising drafts is to identify the devisive issues prior to the f2f
meetings so that there is a chance that they can be worked out in
person.  Sometimes a chair would do better to concentrate on getting
the issues on the table (and using the f2f meeting to discuss those
issues) rather than by merely insisting that the document
be revised before the f2f meeting deadline.


>  - despite IETF mantra that the WG is the mailing list, mailing lists
>    are become increasingly ineffective at resolving issues, e.g., too
>    many postings for many to keep up with, too many off-topic posts
>    (i.e, those not directly related to getting wording changed in a WG
>    document)

These are not equivalent.  maybe you meant e.g. rather than i.e. ?

>   , folk being driven away because small number of vocal posters, etc.

I agree that this is an issue, but it needs more refinement.  There's
an important difference between a list where a small number of people
are exchanging a lot of messages but contributing useful insights while
doing  so, and a list where the level of traffic is similar but the
participants are not making much progress.  (either one might drive some
people away through mere volume, but that's not entirely a bad thing.  a
certain level of discussion is necessary to solve any thorny problem,
and artifically slowing down the discussion to reduce the bandwidth
level is not necessarily helpful.)

Part of this problem may be that we're dealing with more difficult 
issues, and there is more often a need to compromise between very 
disparate conflicting interests, than there was in the past.
A lot of people do get turned off by divisive arguments, but that
doesn't mean that those arguments don't need to take place. 
It might help if the group as a whole could gain a better idea of the
needs of the various interests (both inside and outside of the group)
early in its process - and maybe if these interests were documented 
participants might be able to argue more effectively (less heat and more
light) by having a better idea of their adversaries' points-of-view.  

Keith