MINOR ISSUE: Putting auditing before criteria

Harald Tveit Alvestrand harald at alvestrand.no
Fri Jun 13 22:01:53 CEST 2003

--On fredag, juni 13, 2003 14:28:34 -0400 John C Klensin 
<john-ietf at jck.com> wrote:

> In plain English, "quality audits", at their very best, have to do with
> understanding the nature of the mistakes we have made and when those
> mistakes occur in the process; if we are going to do them, we need to
> improve the mechanisms for _learning_ from those mistakes.

hmm... actually you point at something that touches on a lot of 

> My personal preference, I think, would be to get rid of the notion of
> "explicit quality auditing" entirely and to focus this subsection on the
> need for processes that iteratively improve our criteria for success and
> for measuring progress and for mechanisms for measuring performance (of
> WGs, WG Chairs, Editors, ADs, etc) against those criteria.
> As an example (possibly pointing toward solutions, but intended only to
> illustrate what I'm talking about), the IESG often has a clear sense of
> WGs that have gone very well and others that have gone badly.  Sometimes
> the Chairs of those WGs share that perspective, sometimes they rate
> things differently.  But it is, I think, rare for an AD to sit down with
> the leadership of a WG that has reached its end (or earlier) for an
> in-depth conversation and analysis of what went well, what went poorly,
> and what can be learned about how to better facilitate WG work in the
> future.  I think that is a problem.  But it isn't about "quality audits"
> as that term is used by professionals in those fields.

I left this long quote in because I think it illustrates a point....

there are really multiple purposes at work, at multiple different levels.

One limitation/advantage of our WG process is that it tends to bite off 
reasonable-sized chunks of problem space, chew on them for a while, and 
spit out documents. *Inside* that process, the purpose of auditing, reviews 
or whatever we choose to call it is to make sure the result comes out right 
- and the watchword seems to be "early clue injection" and "no late 
surprises"; the review process needs to be rapidly fed back to chairs, WGs 
and document editors, and written reviews are just so many scraps of 
electronic paper.

However, the WG is always a component in a much *larger* process, dealing 
with the formation, relation, dissolution and retargeting of working groups 
and activities - and it is that landscape of activity that at the moment 
has very little explicit management at any level between the IESG and the 
WG. It is at this level that "learning from the mistakes of the previous 
WG" comes in - not so much as feedback to *this* WG, but as a lesson in 
what to do or not do next time.

It's a level where I believe the IETF has been flailing for a long time - 
the phenomenon of "permanent area-like WGs" is one symptom of one attempt 
to solve the problem without being explicit about it.

At this level, recording history may become important - storing not only 
reviews, but the drafts they were commenting on, as examples of what to 
avoid and why not.... (if anyone ever bothers reading them.....)?
Because the purpose of this review is not to correct *this* action; it is 
to guide the *next* action.

An entirely different kettle of fish.

More information about the Problem-statement mailing list