Sampling

[Dave Crocker]

Folks,

The latter part of a posting by John cites statistic issues.  I'd like
to explore that issue separately and carefully:


JCK> I think it is interesting, and _very_ important, to try to
JCK> understand why the impression of the sentiment of the Problem WG 
JCK> and the response from the Plenary were so different.

JCK> (1) Regardless of the _authority_ of the plenary, significantly
JCK> more IETF participants were sitting there, raising hands, and 
JCK> generally expressing opinions than have been participating 
JCK> actively and observably in the Problem WG.  Unless there is 
JCK> evidence that the plenary was somehow "packed" --and I saw 

I believe that both the working group participation and the IETF Vienna
plenary presentation represent very, very highly biased samples of the
IETF population.

Sampling bias does not require intentional "packing". It requires
selection methods that make the group unrepresentative of the total
population.

In this case, we had a smaller-than-expected total IETF meeting
attendance, in a venue that prevented significant numbers of regular
participants from attending. And not all of the meeting attendees were
Plenary attendees.

Notably, we have not paid much enough attention to just how small the
participation in the Problems working group has been. And the recent
Plenary had all sorts of confusion to this discussion. Overall, we need
to be very circumspect in making decisions based on either bits of
input.

So it is not in the least surprising that the two, differently biased
samples would produce different results.


JCK> This has created a statistical bias between the composition and
JCK> opinions of the self-selected WG and that of the community.

yes, but...


JCK> In other words, the WG has become an island, having discussions and
JCK> reaching conclusions that are out of touch with the IETF universe.

In all likelihood, so was the Plenary.  At the least, the discussion was
confused and confusing.


JCK> 	Sometimes, because of its composition, a
...

Overall, a nicely assessed and written statement, I think.

However the call for Last Call review can (and does) lead to
re-synchronization far too late, I believe. We need to make sure that
there are ways of doing synchronization with the community early and
often.

Even waiting for a Plenary presentation might be too infrequently.
Perhaps we need tight, summary "presentations" on the Announce-list,
occasionally.  We might think of these as "phase" Last Calls, or
somesuch.

d/
--
 Dave Crocker <mailto:dcrocker at brandenburg.com>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>