MINOR ISSUE: Putting auditing before criteria

John C Klensin john-ietf at jck.com
Fri Jun 13 15:28:34 CEST 2003

--On Thursday, 12 June, 2003 22:47 +0200 Harald Tveit Alvestrand 
<harald at alvestrand.no> wrote:

> Section 2.2 of -issue- reads:
>    Some of the key areas where the IETF's practices appear to
> need    tightening up include:
>    o  Lack of explicit quality auditing throughout the
> standards       development process.
>    o  Lack of written guidelines or templates for the content
> of       documents (as opposed to the overall layout) and
> matching lists of       review criteria.
>    o  Poorly defined success criteria for WGs and individual
> documents.
> ISSUE: Quality auditing can only be done by auditing to
> criteria or guidelines. You can't make good guidelines without
> knowing your success criteria. SUGGESTED RESOLUTION: Swap the
> first and third bullets in the list, and change "quality
> auditing" to "auditing against criteria for success". Might
> want to reword the sequence so that "quality" still appears.


I think this suggestion is correct, but want to caution about 
another aspect of it.  In the "quality improvement" and "quality 
assurance" communities, a distinction is often made between 
those activities and "quality auditing".  The cynics suggest 
that quality auditing is about being able to identify the point 
at which something went wrong, sometimes to more efficiently 
cast blame, but not about either better quality or guarantees of 

Unless we have a rather specific plan about what to do with the 
audits and how to incorporate their results into an improvement 
process, such audits are likely to rapidly deteriorate into 
meaningless, time-wasting, bureaucracy and improvements in 
blame-casting processes.   We don't, IMO, need either of those.

So, if your change is made, I suggest that a fourth bullet is 
needed, which might read something like:

	o Lack of adequate processes for feeding the results of
	quality audits forward into process improvements and
	success criteria.

In plain English, "quality audits", at their very best, have to 
do with understanding the nature of the mistakes we have made 
and when those mistakes occur in the process; if we are going to 
do them, we need to improve the mechanisms for _learning_ from 
those mistakes.

My personal preference, I think, would be to get rid of the 
notion of "explicit quality auditing" entirely and to focus this 
subsection on the need for processes that iteratively improve 
our criteria for success and for measuring progress and for 
mechanisms for measuring performance (of WGs, WG Chairs, 
Editors, ADs, etc) against those criteria.

As an example (possibly pointing toward solutions, but intended 
only to illustrate what I'm talking about), the IESG often has a 
clear sense of WGs that have gone very well and others that have 
gone badly.  Sometimes the Chairs of those WGs share that 
perspective, sometimes they rate things differently.  But it is, 
I think, rare for an AD to sit down with the leadership of a WG 
that has reached its end (or earlier) for an in-depth 
conversation and analysis of what went well, what went poorly, 
and what can be learned about how to better facilitate WG work 
in the future.  I think that is a problem.  But it isn't about 
"quality audits" as that term is used by professionals in those 


More information about the Problem-statement mailing list