Latency

[James Kempf]

Henning,

This is an excellent analysis of the problem.

When the move to requirements and framework initially began, I was initially in
support of it, because it seemed like it would help to ensure a better outcome
and allow agreement so that the working group wouldn't be faced with multiple
proposals for protocol designs. But after having had some experience with
requirements, my feeling is that, exactly as you describe, it is just an excuse
to push back the quibbling and second guessing that goes with any group effort
into the requirements phase, then let that surface again during the actual
protocol design when those who don't get what they want during the requirements
bring up exactly the same issues during the protocol design phase. One working
group with which I am associated has spent about 2 years working on requirements
and has yet to get a requirements RFC through the IESG. They are now forging
ahead on the protocol design, and many of the same issues that came up during
the requirements are being brought up again during the protocol design. Nothing
is ever settled until the IESG rules on it and is a standards track RFC.
Requirements don't really mean anything, at best, they are a learning tool for
exploring the problem. They are useful for that purpose, if the problem is ill
defined, but many problems in the IETF don't need that.

            jak


----- Original Message -----
From: "Henning Schulzrinne" <hgs at cs.columbia.edu>
To: <problem-statement at alvestrand.no>
Sent: Saturday, February 01, 2003 7:34 AM
Subject: Latency


> Among many other limitations of the -00-to-RFC data is a more
> macro-scale one that I don't think has been discussed: in many working
> groups that are developing new protocols, the development model has
> changed. (This does not tend to affect our staple working groups that
> are effectively in protocol maintenance mode.)
>
> If we take the software engineering analogy, the IETF has basically gone
> from 'rapid prototyping' to the classical waterfall model
> (http://c2.com/cgi/wiki?WaterFall). Almost any working group has to now
> go through a much longer chartering process, fairly extensive
> 'requirements' and 'framework' documents, before the working group is
> even allowed to talk about protocol development. This is a process
> change which was (I believe) never really discussed and isn't really
> documented.
>
> Thus, I think the perception of long latency is not merely an issue of
> publication delay but also that the whole lifecycle takes longer since
> it is now composed of multiple such publication cycles.
>
> There may well be reasons for this approach, but it might be helpful to
> weigh its costs and benefits. The benefit is presumably better output -
> the resulting work looks much different (and better) than it would have
> without this effort. It would be nice to have concrete examples for that
> (e.g., by asking the authors of the final spec how much the requirements
> helped).
>
> The costs are also clear:
>
> - Additional IESG and RFC editor effort in reviewing documents;
>
> - Exhaustion of technical contributors that spend all their time writing
> requirements
>
> - Different contributors: writing requirements may tend to attract the
> professional standards-goers rather than academics, researchers and
> developers.
>
> - Given the average -00 to RFC delay of at least a year, this adds at
> least a year to the time-to-completion of the real output, not counting
> secondary effects (time spent writing and editing requirements document
> isn't available for other tasks).
>
> - Additional opportunities for obstructionism. It is easy to add ever
> more requirements or argue, in the abstract, about requirements,
> particularly if people are trying to 'position' their prospective solution.
>
> - Delayed failure: Rather than realizing early on that the working group
> is hopelessly divided among competing approaches, the requirements
> documents papers over this difference until a year+ later or simply
> turns it into winning-the-argument-by-exhaustion model of operation.
>
> - Arguments as to whether any actual work follows the letter or the
> spirit of the requirement, even though the particular engineering
> trade-off may not have been foreseen at the time the requirement was
> written.
>
> The last problem is well-known from the waterfall model:
>
> "The disadvantage of waterfall development is that it does not allow for
> much reflection or revision. Once an application is in the testing
> stage, it is very difficult to go back and change something that was not
> well-thought out in the concept stage."
> http://searchvb.techtarget.com/sDefinition/0,,sid8_gci519580,00.html
>
> Thus, a pure 'time to first RFC' metric falls short. I wouldn't consider
> a WG that spent 24 months producing a requirements document a success,
> even if they satisfied some quantitative criteria for timeliness.
>
> To put it less subtly, in some cases we have gone from "rough consensus
> and working code" to "fine requirements and forbidden code".
>
> This may still be the right model, but we might want to reflect on this,
> rather than making it, without questioning, the model for new efforts.
>
> Henning
>
>