More detailed comments on the Problem Statement document

[Elwyn Davies]

Hi, again.

Thoughts below.

> -----Original Message-----
> From: Charlie Perkins [mailto:charliep at iprg.nokia.com] 
> Sent: 07 October 2003 03:09
> To: Problem Statement Working Group
> Subject: More detailed comments on the Problem Statement document
> 
> 
> 
> Hello again folks,
> 
> In general, the document is quite verbose.  A spell checker should
> be employed, and ways should be found to eliminate redundancy.
Apologies for the spelling drop-offs - they will be fixed.
Jeanette Hofmann has already commented on the issue of redundancy (I don't
think there is
much anyway).
As regards verbosity - it is sometimes difficult to avoid touching peoples
hot buttons without a few extra words.
> 
> Here are some more detailed comments on the draft.
> 
> In section 2.1, I do not like the following sentence:
> 
> >    o  The misty vision has inhibited the development of 
> roadmaps that
> >       would inform the IETF's stakeholders of our longer term
> >       intentions,
> 
> In the first place, I don't think the IETF has many longer 
> term intentions,
> except perhaps "saving the Internet".  Such intentions don't 
> necessarily
> need "roadmaps".  I think the IETF is much more reactive than would
> be understood from the above sentence, and I think it's a 
> good thing for
> the IETF to be that way.  This is not to say that the IETF 
> can only solve
> problems after they occur.  I do think think that the IETF 
> should try to
> prevent problems, as far in advance as we reasonably can.  But the
> IETF actions are happening in response to identified problems, not
> (in general!) according to some over-arching roadmap, architecture,
> plan, or vision.

This is a matter of opinion - some people certainly perceive that we need
more in the way of forward planning against a roadmap and others would like
to know further in advance where we think we are going.  Maybe the IETF is
too reactive!

> 
> In section 2.2, on page six, there is the following text:
> 
> >    o  Failure to identify and articulate engineering 
> trade-offs that may
> >       be needed to meet the deadlines that the WG has set without
> >       inappropriately reducing the 'fitness for purpose' for the
> >       intended customers.
> >
> >    o  Continued refinement of the solution beyond the point 
> at which it
> >       is adequate to meet the requirements placed on it by 
> the intended
> >       purpose.
> 
> 
> I am concerned that listing these as bullets needing attention is
> counterproductive.  For one thing, most engineers are aware that
> these problems need to be avoided.  On the general assumption that
> most WG staff (i.e., the ones that do the work and that care) are
> engineers, there isn't much need to restate the obvious, and most
> people will not derive much useful content from these bullets -- i.e.,
> it's the negative equivalent of "motherhood and apple pie".  I don't
> remember when anyone consciously acted in the above bad ways.
> 
I fear I have taken part in a number of working groups that have lost sight
of these engineering principles even if the individual engineers are indeed
well aware of them (the tail end of DiffServ comes to mind - the discussions
of the minutiae of how Expedited Forwarding might be refined in particular).
This is certainly related to the demise of the true purpose of Proposed
Standards and the rise of the thought experiment. 

> In the next list:
> 
> >    o  Poorly defined success criteria for WGs and 
> individual documents.
> 
> I'm not sure this is really worth calling out as a specific problem.
> For one thing, the success criteria are the charter items.  
> For another
> thing, if there is a poorly defined criterion, it stems more from the
> problem with the standards track than anything else, and that's
> not going to get fixed by rewhacking WG processes.
Most charters specify that they will produce a set of documents for
standardizing something (OK some of them do rather better recently) but we
really ought to be clearer about when we will know we have the right answer.

> 
> To the point:
> 
> >    o  Lack of review, especially early review, by reviewers 
> who are not
> >       directly interested mebers of the WG, and by subject matter
> >       experts for topics related to, but not necessarily 
> the immediate
> >       focus of the document.
> 
> I would like to suggest that we not view this as a "problem", 
> because there
> might not be a good solution.  Good external reviewers don't 
> grow on trees,
> and if they did anyway we would find the trees are already completely
> plucked for other reasons.  I don't see how we can expect to 
> get so many
> good external reviewers that they could spend time reading 
> early drafts that
> aren't even implemented by WG members yet.  I'd much rather see such a
> valuable resource be put to use after some initial validation 
> within the 
> working
> group itself.

I am sorry to say that I think this is defeatist and an attempt to brush
under the carpet something which can cause us lots of problems downstream.
I thought that our experiences in software engineering had demonstrated that
early review paid dividends many times over!  Early review needn't
necessarily be terribly time consuming (after all the details are probably
not yet fleshed out) but an experienced SME might well catch the flaws
quickly and before they become too well embedded into the fundamental
structure.

> 
> Another way to attack this deficiency, as I have tried to 
> suggest before, is
> that as time goes on, we amass a body of documentation by 
> subject matter
> experts that explain how to avoid certain cross-working-group 
> problems.
> It's a lot more scalable to ask engineers to try to find 
> relevant documents
> that have been written about a relevant area of cross-expertise 
> interactions,
> than it is for them to find an expert to explain it for the millionth 
> time on the
> expert's personal timeline.
> 
> So, I'd replace that bullet by something like:
> 
> o  Lack of documentation about likely problem areas that might arise
>     due to interactions with other popular IETF protocols.

I'd rather add this bullet - I think you raise a good point but ultimately
there is no substitute for a practised eye and a broad knowledge of what is
going on now (as opposed to distilled wisdom which will inevitably be
abstracted and somewhat out of date).

> 
> The point, on page 12
> 
> >    o  Absence of documented debriefing sessions to assess 
> and record the
> >       successes and failures of WGs (and other processes) 
> so that the
> >       positive and negative lessons learned can be used to 
> facilitate
> >       future work and improve the processes.
> 
> is essentially the same as the previous point, and both could 
> be expressed
> more succinctly as:
> 
> o  Lack of a WG "post mortem" procedure to drive the improvement of
>     the standards development process.

Agreed but it should also extend beyond WG processes.

> 
> The following point:
> 
> >    o  Lack of criteria for determining when a piece of work is
> >       overrunning and/or is unlikely to be concluded 
> successfully either
> >       at all or within an acceptable time frame: Lack of process for
> >       either extending the time frame, adjusting the scope or
> >       terminating the work item or the whole Working Group.
> 
> is debatable, because the process should be "amending the charter",
> and the criteria should be "according to the charter".

It may be debatable but it is a perceived problem - this one of those where
some people see it as a problem but you don't.

> 
> On a positive note, I wholeheartedly agree with the following point,
> and I suggest that current efforts towards improvement be encouraged
> and extended:
> 
> >    o  Automated tools to support the engineering process 
> are minimal.

Phew!

> 
> Regarding the next bullet item, I would suggest that no 
> change is needed
> to current IETF processes:
> 
> >    o  Despite its commitment to 'running code', the IETF is not
> >       proactive in providing ways for developers to verify their
> >       implementations of IETF standards.
> 
> So far as I can tell, the developers are quite adept at making this
> happen regardless of whether the IETF helps or not.  Or, as they
> say, "If it's not broke, don't fix it".
> 
> Also on page 12, the following bullets are redundant and 
> could beneficially
> be deleted to avoid numbing repetition:
> 
> >    o  Project entry, goal setting, dependency identification,
> >       coordination and tracking processes are all either missing or
> >       implemented less effectively than the norm for commercial
> >       organizations in related activities. Dependencies and 
> coordination
> >       should cover both other WGs within the IETF and any 
> outside SDO
> >       with which the IETF is collaborating.
> >
> >    o  Charters regularly fail to set sufficiently granular 
> milestones at
> >       which progress of WGs, individuals and documents can 
> be evaluated.
> >       Also WGs often do not make more detailed work plans 
> to refine the
> >       charter plans.
> >
> >    o  The acceptable deadlines for finishing a piece of 
> work, and the
> >       criteria used to determine them, are rarely, if ever, 
> documented.
> >       Also the estimates of the time required to complete 
> the work often
> >               .......
> 
> Unless I am missing something, all of these ideas have 
> already appeared
> earlier in the section.

I agree that there is some overlap but I would argue that it is justified
because the evidence is posted regarding separate issues (lack of process
improvement techniques and poor application of management techniques).  Also
I think mind-numbing is over-egging the pudding!  If you want mind-numbing
go and re-read the problem statement mailing list archives.

> 
> On the next page, I disagree with the following statement:
> 
> >    One problem which the IETF does not appear to suffer from is
> >    excessive bureaucracy, in the sense that transfer of 
> information is
> >    generally kept to the minimum necessary to accomplish the task. 
> 
> For one thing, I might suggest that the current "rfc-editor" 
> processing
> is quite bureaucratic, regardless of its value.  Also, the 
> evaluation of
> criteria for advancement along the standards track, the submission
> process for Internet Drafts, the BOF approval, and other examples
> can be viewed as more or less bureaucratic.  Maybe it's not as
> bureaucratic as other organizations.  Maybe.
> 
> Another problem is that one person's "minimality" may well be another
> person's "insufficiency".  Typically, bureaucracy creates 
> procedure based
> on historical problem solving episodes.  I can think about the IANA
> considerations in exactly this way -- a little bit along the 
> way towards
> how mortgage agreements are structured.  Of course, one wants to
> avoid ALL systemic problems in permanent transactions.
> 
> So, I'd reduce the amount of self-congratulation on this 
> point.  The best
> you can say is that the IETF bureaucracy is younger than some of the
> other bureaucracies we might be able to name.

A matter of opinion - I haven't seen any other criticism of this point.

> 
> In section 2.3, the first paragraph should cross-reference section 2.2
> since the topic is related to the lack of external review.  
> Perhaps better,
> the bullet in section 2.2 should be moved here, on the theory 
> that this
> is a fact of life, not a problem we can try to solve.

Not everybody agrees that this can't be solved. The cross reference may be
useful.

> 
> I have another thought about section 2.3 that I have moved into
> a separate note.

I have reviewed this and the responses - I think Brian Carpenter's view that
the matter is already covered is correct.  It is however a major potential
roadblock if it results in reinvention of wheels and should certainly be
addressed in 'solutions'.

> 
> Regarding the point on pg. 14:
> 
> >    o  The IETF does not have an effective means for defining
> >       architectures and frameworks that will shape the work 
> of multiple
> >       WGs.
> 
> ... that is something that should be done by the IAB.  So I would say
> that the IETF does have the means, but it's perhaps not yet effective.

Maybe it ought to but it is not clear that it believes it has a mandate to
do this.

> 
> On page 16, in section 2.5, the following sentence should be deleted:
> 
> >    Whilst this might be put down to contributors having less time
> >    available in their work schedule during the downturn of 
> the last two
> >    years, this is not the whole story because there were 
> signs of this
> >    effect back at the height of the boom in 2000.
> 

I disagree. I think that it is important that this is not just dismissed as
a side-effect of the industry downturn.

> 
> In section 2.6.4, the last paragraph should be deleted.  It 
> has no place,
> and the most charitable interpretation would put it squarely into a 
> solutions
> document.  More likely, it looks like an insult.

This paragraph has escaped the de-solutioning sweep.  However I don't
believe it should be deleted.  It certainly appears that in some cases IESG
delays have been used as a smoke-screen to disguise or even foment slow
progress (where some interest group doesn't really want progress).  I'll
rephrase it a bit.
> 
> In section 2.7, the claim is made that problems result from the
> non-hierarchical nature of the working group.  I don't believe it.
> I have never seen this as a problem.  In fact, I would characterize
> it as a feature.
> 
> If anything, there might be a problem with assignment of tasks,
> and accountability for accomplishing those tasks.  Typically,
> hierarchy helps with delegation, accountability, and evaluation,
> but the lack of hierarchy does not mean the lack of those
> three management aids.  In fact, I would in solution space
> suggest that the new methods for issue tracking could be tied
> in with a non-hierarchical method for delegation, accountability,
> and evaluation for better results.

It is certainly a feature but many people find it difficult or contrary to
their normal working experience, and it certainly can make it very difficult
to handle deadlines.  I am guilty as charged!

Regards,
Elwyn.

> 
> I will send out yet another note with detailed editorial remarks,
> with a lot less substance and importance compared to these
> comments.
> 
> Regards,
> Charlie P.
> 
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://eikenes.alvestrand.no/pipermail/problem-statement/attachments/20031017/5706386e/attachment-0001.htm