Document: draft-ietf-bmwg-ospfconv-applicability-07.txt draft-ietf-bmwg-ospfconv-term-10.txt draft-ietf-bmwg-ospfconv-intraarea-10.txt Review: Brian E Carpenter Date: 13 oktober 2004 Considerations When Using Basic OSPF Convergence Benchmarks (Informational) draft-ietf-bmwg-ospfconv-applicability-07.txt OSPF Benchmarking Terminology and Concepts (Informational) draft-ietf-bmwg-ospfconv-term-10.txt Benchmarking Basic OSPF Single Router Control Plane Convergence (Informational) draft-ietf-bmwg-ospfconv-intraarea-10.txt Good quality documents ready for publication. I do have one question of principle: do we need a stronger disclaimer? (See below.) The best order to read these drafts is 1) -term- 2) -intraarea- 3) -applicability-. The pitfall is that the topic is performance of a *single* router in specified network topologies, which of course doesn't say much about the convergence time of a real network. There's a risk of this methodology being used to compare apples and oranges, e.g. (from -intraarea-) for the basic SPF convergence time measurement: > Note: This test may not be accurate on systems which implement OSPF > as a multithreaded process, where the flooding takes place in a > separate process (or on a different processor) than shortest path > first computations. Another example is section 6 of -applicability- (on the impact of the exact topology used for SPF convergence time measurement). So should we be concerned about these documents being used for RFP evaluations? I just wonder whether a stronger disclaimer is needed, in addition to this at the end of section 3 of -applicability-: > o The measurements described in [BENCHMARK] should be used with > great care when comparing two different implementations of OSPF > from two different vendors. For instance, there are many other > factors than convergence speed that need to be taken into con- > sideration when comparing different vendor's products, and it's > difficult to align the resources available on one device to the > resources available on another device. Nit in -applicability- section 4.2: > The DUT also adds noise to the measurement. Points (a) and (c) > apply to the DUT as well. There are no points (a) and (c); the bullets are not labelled.