The purpose of the IETF
This is based on an IESG retreat held just before the London IETF, August
2001, with some connection to other sources.
The purpose of the IETF is to create high quality,
relevant standards for the Internet
The common purpose that brings the IETF together is that "The Internet
We have a shared vision of a common, ubiquitous, open networking infrastructure
that can be used for a multitude of purposes, bringing the benefits of
communication to bear on a variety of tasks.
A large part of making the Internet work is the job of others, and
NOT IETF business - because others do it better.
However, we seem to have a common view that no other body is doing a better
job of creating standards for the "core of the Internet".
Deploying networks. ISPs do that.
Building products. Vendors do that.
Regulating the ways in which the Internet can be used. Governments do that
(and ICANN tries to).
We have a number of limitations in the standards area too:
But - there is an area "in the middle" that is not covered adequately by
any other standards organization, and for which we believe that the IETF
has unique expertise that it can bring to bear on the problem.
The IEEE seems to do a good job of specifying Ethernet-like technologies
The ITU seems to have a handle on specifying physical interconnects
The W3C is welcome to the problems involved in designing document formats
The borders of this area are intentionally fuzzy - if something
turns out to be a real problem that needs an Internet-based solution, we
want to be able to bring our expertise to bear on it - IF that is of benefit
to the Internet.
This "warm and fuzzy" does not translate real easily into measurable
work items. But we can try.
Metrics for IETF success
In any situation where you need to know whether you are doing good or doing
bad, you have to measure. And what you measure will have a direct influence
on what you optimize for. Therefore, the metrics to measure the IETF matter.
Volume of output - number of RFCs, number of pages - is a BAD metric.
It says nothing about quality or relevance. Some have argued that reducing
the number of standards would be a measure of success.
Usefulness of output - very hard to measure directly.
Speed - as "WG proposal to finished RFC" - contributes directly
to relevance, because it is easier to design a relevant solution when the
base conditions don't change too much between initiation and finished product.
However, excessive focus on speed is likely to impinge on the quality goal.
Adoption in product - a measure related to "usefulness". One should,
though, be careful of the "tick box effect" - people will ship a bad/incomplete/useless
implementation of a protocol just to get "the mark", knowing full well
that it is not going to be used. This is not contributing.
Adoption in use - more closely related to "usefulness". Takes time
to happen, and is hard to measure. But important.
Clarity of specification - a clear specification is more likely
to be implemented correctly.
Shortness of specification - terseness CAN contribute to clarity.
It can also do the opposite.
Cruft resistance - many options and add-on features reduce quality.
Product interoperability - very important; the IETF is about communication.
Metrics for IESG success
At the "high order bit", the IESG succeeds if the IETF succeeds. We must
then look at what the IESG can do to contribute to the IETF metrics, and
see if there are metrics that we can use to see if the IESG is succeeding.
A problem in the first 2 bullets is that we have a quality control problem
- if the WG proposals or the WG consensus documents are not high quality,
speed will drop like a stone - and SHOULD. We need metrics to capture this,
WG creation speed - time from proposal to WG chartered
Review cycle speed - time from "WG declares consensus" to "sent
to RFC editor"
Management quality - degree to which the ADs are helping WGs in
progressing towards their goals
Bloody-mindedness - efficiently getting rid of efforts that are
not contributing to the goals
Coverage - making sure efforts that are needed for the Internet
get started when needed
No lost tokens - nothing should be forgotten, even if delayed.