Draft minutes from San Francisco

[Melinda Shore]

I've appended a first draft of the minutes from the San
Francisco meeting.  I've kept them more wordy than usual in
the interest of keeping as much information available as
possible.  Many thanks to Spencer Dawkins and Chris Allen
for their detailed notes, and to Lisa Dussault and Ted
Hardie for their scribe work in the Jabber chat room.

Please send corrections, etc., to the mailing list.

Melinda



Problem Statement BOF (problem)

Friday, March 21 at 0900-1130
==============================

CHAIR:	Melinda Shore <mshore at cisco.com>
	Avri Doria <avri at apocalypse.org>

Notes:	Spencer Dawkins <spencer at mcrs-labs.org>, Chris Allen
    <ChristopherA at AlacrityManagement.com> 

Charter discussion - Chairs
	Charter went through quickly with little controversy
	work is to be analytic and descriptive, with emphasis on analytic
	not pointing fingers
	aggressive schedule - done in October
	comments on document structure and contents both welcome
	
Current Status
	Mailing list but no separate web page - do we need one?
	1st draft of problem description document has been released
	Have editors and editor team for both chartered documents

Chris Allen: suggested addition to charter - document
existing procedures to keep what's effective

Document Process Performance - Henning Schulzrinne

Identified "IETF Goodput Measures": size, delay, throughput.
This was a first attempt to figure out if this is worth
looking at.  Methodology: matched RFC names with draft-00
titles, looking for "end-to-end latency."  Our data are
extremely poor - no other mechanism works.  The Internet
Monthly Report doesn't contain all I-D names and the IETF
Announcement list only goes back to 1998 so history has been
lost.  Help is welcome on this.

Average number of pages spiked a lot earlier than number of
RFCs per year. It hasn't changed since early 80s -- about
200 RFCs per year, about 25-30 pages per RFC.  Internet
draft bandwidth has been on steady linear increase since
1991.  2001 was down, 2002 was back up.  00-to-RFC delay has
grown from 6 months to about two years, with a steady
increase from 1991 to 1998, and then constant since then.
Harald Alvestrand asked if he'd tried plotting the graph
using the year the work was started instead of the year the
work was ended.  Henning said he hadn't.  In response to a
question from Chris Allen Henning said that many measures
actually fell off somewhat in 2001, not just these.  Several
people suggested additional metrics and Henning responded
that other statistics are also interesting

The RFC Delay Histogram clusters at two years, but has a
long tail - four, even five years.  Aaron Falk asked about
individual submissions that become working group document,
and Henning responded that the stats were based on title
matching, not I-D name matching.

The data are VERY noisy - formats highly variable, Internet
Monthly Report is incomplete, some RFCs were never
announced.  The estimates are likely low due to I-D
renaming or high due to "blocking."

Some suggestions: sample, don't census
	Look at questions like "one rev per IETF meeting"
	Waiting for other documents? author fixes? IESG queues?
	Look at reasons, not at data
Does this help agreement on root causes?


Initial Analysis of IESG Review - Eric Rescola

Overlaps Henning's analysis.  He also experienced low
signal-noise ratio, but basically correct.
The question being address was how long do things take from
last call to document approval.

What he found was:
	mean = 147 days, median = 93 days - standard deviation is 149 days
	300 days isn't unusual
	Documents that are published as is are still slow
	Areas differ a bunch in speed:
		internet, Ops and transport are the fast ones
		applications and security are the slow ones
	and differ a bunch in output volume

Churning takes time (60 days, volume of changes doesn't
matter, length doesn't matter.  The approval time is
constant over study period.

Dave Crocker said this work was similar to work Marshall and
he did and that there were some methodological problems with
the treatment of the data.  he said that he was not saying
the math was wrong, but use of those statistics in that
environment is risky because the community of professionals
use normal data, but don't know how to process log-normal
data. There is a difference between the math world and the
social research world.  Eric disagreed about the
appropriateness of the tests and said that the methodology
was fine.

Bill Sommerfield said that as a WG chair he's frustrated.
In his working group they get one bug report, they fix it,
submit, get one more bug report, fix it, etc.  Are there
data on small changes?  Eric answered from 0 to 4 review
cycles typically.

Aaron Falk said that AD review isn't well documented and can
be highly variable, and Eric replied that it can be highly
variable between ADs, as well.


Architectural and Process Inputs - James Kempf

James's discussion was titled "Helping the Internet to age
gracefully." 

Heard around IETF: "We don't do any architecture," with
architectural work hidden as "frameworks", etc.  Heard
around 3GPP - "IETF doesn't do architecture, they do
philosophy."

End-to-end is an example of traditional approach and it's
helpful, but we need more.  We must integrate new components
with existing components.  There are interdependencies
across working groups, or even across areas.  Catching them
at IESG review time is too late to affect base design.  Is
this really system design, and not architecture?

IETF work is a stove pipe, and assumes minimal
interdependencies - but the number isn't zero.  Can we
ignore the problem?  We're sending more drafts back due to
conflicts, but missed interdepencies still happen.  There's
an increasing lack of transparency and understandability in
Internet Design.  It's becoming more like the telephony
world - baroque.

Bob Hinden raised the issue of proposed standards and said
that there are risks in implementing PS, but they implement
PS in products all the time.  James replied but the problem
might be big - not a minor tweak, and Bob pointed out that:
full standards are really rare.  Bill Sommerfeld: too
difficult to move past PS - there's no energy after doc is
stable.

Henning said that he doen't disagree with slides, but needs
case studies for background, with concrete examples of
last-minute/too-late fixes.  James: this would be useful.

	
Working Group Participation and the "Stuckee Problem" - Ted Hardie

The discussion is abstracted from and I-D that's
solutions-based.  The IETF working style of an Open working
group on mailing lists has produced successful results, but
making a comment doesn't mean you're taking responsibility
for doing work.  It's difficult to make anyone other than WG
chairs and current authors accountable.  The problem is how
to track commitment without breaking openness and shutting
out outside review.

Dave Crocker said that we have people who consume resources
but don't help make process.  Ted said that may be true but
that isn't the problem he's looking at.  Dave said that may
be theoretically true, but isn't this a test for working
groups getting formed?

Ted answered that working groups do vary over time, but
stuff does fall out the other end.  There may not be a
constituency when it does, but that's not what we are
discussing here.  The problem is that we can't do anything
that has dependencies unless someone is actually responsible
for those dependencies.  Spencer Dawkins said that
dependencies are worse with other SDOs (3GPP).

How can working group chairs responsibly say "you can count
on us?  Bob Hinden said that if there's no energy, the work
shouldn't be done, and Ted replied that if another working
group is depending on the work that's not being done, the
second working group either fails or takes on all the work.


Problem Resolution status - Margaret Wasserman

This is the second document in working group charter.
There are clearly issues about changing the IETF.  Where do
we go from here?  The IETF is unique, but these aren't
unique problems.  Good times hide problems, bad times hide
successes, and we need a consistent focus on operational
improvement.

Margaret listed Harald's core values from IETF 55 plenary.
Things that aren't core values are all on the table for
changes.  The current environment is highly emotional but
we're committed to improvement.  The IETF is a fast-moving
target - we can't stand still while we measure and decide
how to move forward.

Major process options:
	Evolution vs revolution ("nothing left to lose"?)
	Granular vs monolithic (can problems be separated?)
	Immediate versus ongoing
	Grassroots vs top-down

What are the right mechanisms to use, moving forward?
Should we update existing process documents?  Do we need new
roles documents?  We need a initial draft in the next couple
of weeks and haven't gotten much input yet.

James Kempf comment: People are assuming that there is one
solution, and that the entire IETF will adopt it. We should
try a bunch of different approaches and see what ones
work. This means that we have to be willing to accept that
some will not work.  Margaret pointed out that that is one
of the listed solutions.

Harald: Fixing our short-term problems is harder than
fundamental changes to IETF and may be harmful to the
Internet.  If we focus on short-term problems it takes the
heat off for five years, but if we don't fix short-term
solutions, we're also dead.

Dave Crocker: and what will be the experimental group?
Everything we've done for 13 years has been an experiment
revolutionary changes have big and unexpected impacts
	
Bob Hinden: Harald is right - we can't fix our problems by
fine-tuning.

Harald: if we get to the point where we can't declare
consensus, I'll go home.


Problem Statement draft - Elwyn Davies

This is a very provisional version (00 draft).  The problems
aren't new and aren't (all) IETF-specific, and many are the
consequence of growth.

The draft was distilled from 750 pieces of email plus 3
drafts.  The topics were extracted and filed, and in the
case of duplicates they were attributed to the person who
mentioned them first.

The next steps are to create a visible issue list and
publish a second draft by late April, WG last call and IESG
review by May.
	
Aaron: should be commended for your altruism and suspected
for your judgement in taking this role!  However, thedraft
really bothered me - perceptions start to sound like facts.

[Keith Moore objected to the presentation of material that's
already available in the document and that was presented at
the plenary, and asked that we focus on discussion rather
than presentation.]

Dave Crocker responded to Aaron that it doesn't matter that
perceptions sound like facts - we can use this as is.

Graham Travers pointed out that people holding perceptions
is a fact, whether the perception is right or wrong.

Chris Allen: there's a cultural element in section 2.1, and
mission and vision go hand in hand.

Bob Hinden said that two things were mixed in section 2.2 -
WG process and protocol design engineering.  Henning raised
the issue of a lack of effective contract with document
editors, to which Elwyn added that this leads to lack of
delegation.

Margaret: effective engineering processes isn't the goal,
it's a solution, and we need to focus on identifying the problem.
Eric Hickman: over time we apply ourselves less and less,
especially towards achieving consensus in adversarial
working groups.

Keith: lots of 2.2 emphasis on tools and process, and
problem is really more fundamental - groups that can't
explain their problem clearly.  Don't compare 2.3 to "good
old days" - we're just not sufficiently engaged.
Christian: can't stay engaged in SIP at 400 msg/day.
Randy Bush: more people are engaged, but there's more people who
aren't

Nathan: Money has gone out of this industry, and more money to be made
implementing than inventing
Steve Trowbridge: Spectators aren't the problem, people frightened
away is the problem
Eric Guttman: people leave mailing lists because they don't want to be
insulted
James: physical violence threatened on one mailing list
Steve: ADs/WG chairs wait too long to wade in
Spencer: working group code of conduct?
Keith: mailing list and meeting methods are very different - should
the document split them out?
	
Keith totally disagreed with 2.4 as it was written.  It is
clear that there is a perception of this, it is clear that a
perceptual differences are part of the problem, but he
doesn't think it is fact.

Steve Trowbridge said that there is nothing in our
procedures that causes this to be the case, it exists in
practice.  There are backroom deals and initial windows on
work available only to insiders.
Henning: in the good old days, ADs were a LOT more
invisible, now anything of any significance needs AD
involvement. 

Margaret said that this section is where the scaling issues
of the IETF are showing up.  
Jonne Soinenen: no single person is the problem, but the
process has problems and the perception of problems is
equally important as the problems themselves.

Keith commented that no one will disagree with perception
that IESG hasn't scaled, but qualifying it as 'authority' is
inappropriate. He was in IESG 4 years and they really can't
block things. This isn't an accurate statement. He does
agree that it's possible for an AD to be capricious and we
should fix that.  Marshall answered that there are some
former ADs that say that this has happened, in spite of what
Keith said.

Eric Guttman: if openness is our goal, why is the IESG is
deciding WG formation, document advance, etc?  Eve Schooler:
the IETF isn't more broken than it was ten years ago, that's
good news.  People do want to help, and we need to be
grooming more potential leaders/insiders.

Margaret said that there are other problems besides scaling
and asked to what extent do we want rely on WG consensus vs
AD agreement.  Are our selection processes open and fair?
Jonne: we should see documented discussion of decisions so
we can see why they are made.  People just don't know what's
going on.  We shouldn't require people to be on the IESG to
understand the IESG.

Chris Allen: in groupware field ten years ago, social
scientists were looking at mailing lists.  we don't teach
people what's already well-known and undersood about our
work media.

Pete Resnick: the perception that there are upper
class/lower class differences is real.  Distrust is real and
can increase if we're not careful.

Margaret: WG chair role isn't well-defined and consistently
understood in the IETF.  We need to do role definition
before training can be effective.  Chris Allen added that
the mentoring process isn't mentioned and is broken.  We
have document editors and WG chairs who are brand new.  We
should have "shadow" roles who works under old person for 6
months/one year for training.

Bob Hinden: There are problems that come with getting new
people and readying them to be come the new leaders. I have
the feeling that it has become more exclusive. There is a
perception of cycling from IAB to IESG, IESG to WG, etc.

Joel Halperin: ADs CAN block things, and sometimes the
things we're complaining about are the reasons why we
succeed.  Don't create a new problem while solving another
one.

Bob Hinden: "blocking" needs to happen at the beginning of
the process, not at the end.
	
Ted Hardie: Part of the problem is the metaphor "Send up to
the IESG" which colors the perceptions. There are many
things in the document that have a series of metaphors of
consensus, and we use the words of consensus, but in fact,
sometimes we use disputation.  When we use disputation with
consensus it can cause acrimony.  We need metaphors to match
our recommendations.

Dave Crocker said that we may not have structural problems,
but rather management problems.  For instance we sometimes
choose between two alternatives when we don't have to, and
sometimes we don't choose when we should.  Christian:
"Consensus" wasn't in original IETF, but was added in 1992
as response to top-down decision making.  The way we used to
solve problems was by building something.  Now we need
agreement before anyone builds anything.

Eve Schooler: We can say "here's a good Internet Draft", but
"what makes a good WG chair or AD" isn't written down
anywhere.

Ted: we've narrowed the point of WGs until it's hard to see
the whole picture, then we're surprised when we get
problematic point solutions.


Wrapup - chairs

We're serious about hitting our milestones.  The problem
description is to be in WG last call in May, and the process
options document is to be in WG last call in August.  The
latter document should be available within the next few
weeks.