Scribe from Problem-Statement WG Meeting, SF

[Christopher Allen]

Problem-Statement working group meeting

* Blue sheets/Agenda Bashing
 Agenda Review

* Status (chairs)
 Active as of 6 March 2003
 Chairs: Avri Doria <avri at acm.org>, Melinda Shore <mshore at cisco.com>
 Editors & Editing Team (list of people)
 Answer to floor question: Editor team is for both documents

* Charter Discussion (chairs)
 The charter has not been controversial
 The workgroup is to be more analytic then descriptive
 The WG has a very aggressive deliverable schedule, ending in October
 First document to be done in a couple of weeks
 First version of a document is a proposal, not a final
 We will review at IESG plenaries in future meetings
 Christopher Allen: Would like to add to charter "...and we should have a
clear understanding of existing procedures and rules which are effective, so
that we may preserve them."

* Document process performance (Henning Schulzrinne)
 IETF Goodput Measurements: Packets Size, Delay & Throughput
 For our deliverables
  Determine throuput, packetsize, delay in publication of RFCs as a function
of time
 Delay measured from --00 to RFC, by title
 Based on:
  rfcindex.txt, IMR, IETF announce
 Inserted into EDAS/netbib (sql database)
 Data is very poor, very difficult to compute
 Data based on matching RFC title with I-D title
  Only IETF announcement list contains original I-D filename!
  This information is lost! No archive earlier then 1998!
  We have lost our announcement history older then 5 years.
  Lack of this old information is causes some data analysis problems
 RFC Bandwidth and packet size (chart)
  RFCs published per year
   50 or so in 60s
   100 or so in 70s
   begins to flatten and jump
   150-200 or so per yearin 80s, 90s, and 00s
  RFC avg. pages
   Growing pages, but not as much as Henning thought
 I-D Bandwidth
  Though RFCs production is flat, I-Ds are growing fast
  500 in early 90s, 2000 in mid 90s, 3500 in 2001
  Is not significantly increasing since 2001
   Substantiated by other numbers like attendance
  This is not just 00 IDs, but all i-d actions changes are noted
  Hasn't done an analysis of only 00 drafts
 RFC Delay
  Delays in early 90s 5 months, mid-90s 15 months
  since 97 has been roughly 20-25 months
 RFC Delay Histogram
  There is a clustering at 2 years delay, but there are some significant
tails longer then 2, with 5 years.
 Lots of comments from floor about definitions, questions about title
matching, questions of whether all was captured, hopeless of capturing some
info.
 Comment of floor: frequency of updates is a problem
 Caveats from data
  Noisy Data
  Lots of ways of formating I-D announcements
  Lots of editing areas
  IMR is incomplete
   Some RFCs were even never announced!
  The estimate is likely low due to I-D "elevation"
   - draft-alice-problem -> draft-ietf-problem
   - drafts merged
  (lost some stuff from slide, speaking running out of time)
 Suggestions
  Sampling instead of complete statistics
  Could get delay from final I-D to RFC with some effort
  Track biography of random RCS and try to ascertain delay compenents
   - one I-D iteration per IETF
   - waiting for other documents
   long delay waiting for author fixes in RFC editor queue
   IESG "Tabling"
  Dont' care about minor statistical variation
  (lost one item)
 Summary
  Statistics reasonable stable since ~1988
   - Avg. RFC size has not increased since 1990,
  (lost last sumary item due to rush)

* Issues in IESG Review (Eric Rescorla)
 ("something of in the water of a desire to do statistical research", wishes
he had Henning's data)
 Analysis of 4+ years of IESG Review
  IESG Review is an important process
  (lost points due to speed)
 Basic question: How long do things take from last call to approval
  mean 147 days
  median 93
  standard dev 149 days
  long tail of 600 days
 Drafts that are approved as-is (no changes) are faster, but not much
  average is still 78
  mean is 54(? not sure if scribed correctly)
 The areas differ quite a bit in speed
  INT, O&M, and TSV are significantly faster
  Applications and Security are slower
  Rate of approval is about twice as fast from slowest to fastest
 Big diferencs by AD in output volume
 Document Issues
  Churning consumes time
   each revision consumes 58 +/- 6 days
   lots of noice (r-squared=.39)
  But the volume of changes doesn't make any difference
   once you control for number of revisions
   This is rather surprising
  Length of document doesn't matter
   This is also surprising
  Things aren't getting and slower, but not faster
 Comments from floor
  Dave Crocker: There are some problems in methodology, some problems with
normal, massaging data and using standard deviation can cause problems. I'm
not saying the math was wrong, but use of those statistics in that
environment is risky, because the community of professionals use normal
data, but don't know how to process that data. There is a difference between
the math world and the social research world.
  Eric Resc.: Disagrees
  Bill Sommerfield: As a WG chair, I'm frustrating, I get one bug report, we
fix it, submit, get one more bug report, fix it, etc. Do you have data on
small changes
  Eric Resc.: From 0 to 4 review cycles typically
  Aaron Falk: One of the things that gets in the way of understanding that
is because the AD review, other undocumented aspects
  Eric Resc.: One difference between AD groups may because exactly this.

* IETF process and architectural input (James Kempf)
 Presentation title "Helping the Internet to Age Gracefully"
 One of our problems is that the Internet is getting older
 Heard around the IETF
  "We don't do any <insert your favorite explitive> architecture"
  Can't use the word architecture, we have to use framework
 Heard around 3GPP
  IETF does philosopy
 Internet Architecture Historically
  Architecture historically
   Articulation of broad design principals
   Development of conswquestions
  (lost items to to speed)
 What is the problem
  Need some way to maintain consistency
  New componenents must integrate with existing system
  Removing or extending old components must honor interdependecies
  sometimes interdependencies now cross WGs and even Areas
  Catching them at IESG REview is often too late to affect the base design.
   By the time the drafts get the IESG, they can't do much more then throw
back, or only get the worst problems fixed
  The architecture the wrong word?
   Maybe "System Design" (John Klensin)
 Possible Consquences of Ignoring this problem
  More drafts returnted to WG during IESG review due to conflicts
  Occasional missed interdependencies turning up after draft has gone to PS
   - Interoperability problems
   - Delays in successful deployment
   - Problems in products
   - increase expense
  Increased lack of transparency and understanding in Internet design
   - One of the advantages that internet has over telephony world is that it
is relatively easy to understand, not baroque.
 Comment from Robert Hinden(?): Proposed Standard means proposed standard,
it is the experience of those who risk implementation of proposed that
drives to draft standard. Something may cycle at PS several times before
going to DS.
 Bill Sommerfeld: It is so difficult going to proposed standard, that most
people give up after draft standard is done. You are out of energy often
right before Proposed Standard, much less after to move to Draft Standard
 Henning: We need more case studies. There are a lot of particulars lost.
 James comments: agrees that we need more studies, is doing a draft


* Issues in working group structure (Ted Hardie)
 Working Group Participation AKA -- The "Stuckee" problem
  (comments about the amount of time lost making laptops display)
 Intro
  Has written on a draft on 'stuckee's
  Belinda challenged Ted to present it as a "problem"
 What are WG accountable for
  Within charter:
   make good engineering decisions
   specify them
    clearly
    promptly
 Working Group Theory
  Historicaly are defined by mailing list, all decisions require consensus
oflist
  No membership requirements
   tech commetns fromanywhere, cross-fertilization is possible
   specific open calls to all
 Opening vs commitment issue
  making a comment on a document does not imply that you are taking
responsibility for the work of the working group
  That ambiguuity makes it difficult to predition
   How much attention
   estimate time
   estimate effort
  Openness can make it difficult to make anyone other then SG chair and
authors accountable.
 How do we retain openness
  We need commitment to
   Make decisions
   Produce docs
   Review docs for accuracy, readability, and usefullness
  without
   shutting out the technical comments from those outside the WG or Area
 Comments from Dave Crocker: Do you mean that the problem 'people consume
time and bandwidth and don't contribute." I'm not sure what the commitment
step buys. It is true that WGs vary over time, but over time they loose
people and change participation, but thus a loss of constintuancy. If they
don't show up, they don't get to speak: that is the test that makes things
work.
 Reply by Ted: Not precisely the problem that I'm stating, it is more how
can WGs chairs can do both. This ambiguity causes it to be difficult for
estimates for WGs to estimate and predict progress on work, in particularly
as multiple groups had interpendencies. I believe that this needs to be as
much an open process as it can, but that openness is so unbounded over time
cause problems. We want commitments that survive over time, that if you
commit you should follow through.
 Dave Crocker: Everything you said is extremely reasonably, but is very
different then the philosophy and history and vision of the IETF. It may be
that to fix it is to change the statement of the problem. Dave does realize
that since he has spoken up on this topic that he needs to work on it ;-)
 Spencer Dawkins: Other groups (non-IETF groups) also have problems with
interdepencies with the IETF
 Robert Hinden: Sometimes work that is required by another working stops
because the providing group looses energy, whereas the dependent group has
it.

* Process Document (Margaret Wasserman)
 Title: Problem Resolution Process
 Talking about the second deliverable of this group
 Process Document
  A proposal for a process to develop solution to the problems identified
  Describes the process of the process
 Milestones
  Mar 03 First I-D of process document
  Jus 03 Process review at IESG plenary
  Aug 03 Process document submitted to IETF
  I need comments on list on discussions already there
 Document in progress
  Thoughts about change in the IETF
  High-level process considerin
  Mechanisms, knobs, leves
 Probem Perception
  Good time to hide problems
   Tendency to sit back and rest on laurels
  Bad Times Hide Successes
   Tendency to be over-critical and reactive
  Solution
   Consistent focus on operational improvement
   Build a culture to do this
   We need to do this in the IETF
 Core Values (from IESG Plenary 2002)
  Cares for the internet
  technically competent
  oepn process
  volunteer core
  "We reject kings, presidents and voting. We believe in rough consensus and
working code"
 Not core values
  The division into WGS and AReas
  The three-step standards process
  The ASCII format for RFCs and IDS
  The format of IETF meeting
  The way mailing lists works
 Current Environment
  Tension and Emotionalism
   Us vs Them mentality
   Frustration on all sides
   Lack of trust between leaders/non-leaders
  Commitment to improvement
   People on all sides seem committed to improving the IETF
 Moving Target
  The IETF is a moving target
   This make things tricky
   Process of acknowledging/identifyin problems is already creating chagne
   Can't ask the IETF to stand still while we measure it
 Major Process Options
  (I don't know where we stand on these tradeoffs)
  Evolution or Revolution
  Granular or Monolithic
  Deliberate or Immediate
  Grass Roots or Coordinated
 Details
  Evolution
   Focus on iterative improvement
    indentify
    formulation
    reassess
    repeat
   Most appropriate when things are working
  Revolution
   Focus on large scale change
    Determine type and scope of change
   Best when fundemental changes are needed
    Nothing left to loose
  Granularity
   Address separate problems separately
    several smaller efforts
    different process choices
    allow execution at different priorities
   Works if separable problms
  Mololithic
   (lost, too fast)
  Deliberat
   Careful process with several steps
   Best for non emergency probems
    good for complex problems
    need to measure what is good
  Immediate
   Determinte a change and execute
    Base
   Best in an emergency
    avoid analysis paralysis of the deliberate approach
  Grass Roots
   Improvement efforts start from below
    Let those who have an idea propose
    leadership determines which have priority
  Coordinated
   Establish a time responsible to the problem
    Problem space oriented
 Available Mechanismes
  Ask leadership to correct
  Hold a BOF
  Form a WG
  Form a Design Team
  Assign problems to individuals
  Others?
 Knobs and Levers
  Update existing process documents
   RFC 2026 etc.
  Develop new processes and roles
   IESG charter, proceedures
  Organize independent effotts
   Organizational development training
 Where do we go from here?
  Need to develop intitial draft in next few weeks
  Should be responsive to the problems
  What should it say?
  Not much commetns on list.
 James Kempf comment: People are assuming that there is one solution, and
that the entire IETF will adopt it. We should try a bunch of different
approaches and see what ones work. This means that we have to be willing to
accept that some will not work.
 Margaret replies: That is one of the possible solutions above.
 Harald: Paradoxical statement. Fixing our short term problems may be
harmful of the internet. My conclusions are that fundemental problems with
the IETF may cause serious problems of the internet. In five more years that
we may be dead if we don't make fundemental changes. That said, we have to
do some short term fixes! A paradox.
 Question from Floor: Does that mean that fundemental change has broad
consensus?
 Harold: <nods affirmatively>
 Dave Crocker: What will be the experimental group? We have been doing an
experiment for 13 years, we do the best we can, and it works. We don't have
enough controls to unerstand. Effects can be ranged, varied, and variable.
 Bob Hinden: Harald is right, we need to make fundemental changes.
 Harald: How do we reach consensus, if we can't reach it, I will go home.
This is an open process, I need contributions.
 Chairs: Keep talking about this on the list!

* Problem definitions/scoping document (Elywn Davies)
 IETF Problem Statement (discussion of draft)
 Root Cause Approach
  differentiate derivitive problems from primary
  current org of draft is provisional
 Initial thoughts
  The problems are not new
  the problems are not IETF specific
  many are consquense of growth
 First Pass
  750 emails and 3 drafts
  Topics extracted and filed, attributed to first contribution
  Categoried used baseic structure and process of the IETF
   not retrofitted
 Next Steps
  Discussion of breakdown andetails
   Today and continued onlist
   looking for
    alt causes
    better breakdown
    better wording to clarify this is not a witch hunt
   In parallel - creation of Process Recommendation
  (missing some items)
 Initial List of Root causes
  (list from draft)
 Comment from floor: You should be commended for taking this on. HOwever,
Reading draft bothered me. It makes things sound like facts that are
perceptions. I don't feel that way for some of the pieces. That they were
stated as fact is a problem.
 Comment from floor: We don't need to see this list, we saw it in plenary.
Lets not waste the 10 minutes on going over the list again.
 Dave Crocker: I'm part of the editing team, I want to note that Elwyn
turned out that document on his own, and it isn't a thankless job, as we
think we've thanked him a lot. I don't care if it is written as facts, but
it does have value as it is because there is no such thing as 'fact'. We
need to get a rank ordering of consensus. One important issues is that we
need to make some priorities, some things we can do now with big impact,
etc. It is a big matrix of decisions.
 Graham: If there are some people that have a perception of a problem, that
is a fact. Even if they are wrong, the perception is a fact.
 (this scribe refers you to draft for the list of draft)
 On 2.1: Mission
  Christopher Allen: We are not looing at the culture, the mission can be
too dry. What drives people to help are the culture and values
 On 2.2: Effective Engineering Practices
  Robert Hinden: This maybe needs to be broken into two items, project
management and quality management.
  Henning: Lack of granularity of milestones, lack of support, lack of
priorities. We need a 'contract' with the authors.
  Elwyn. There are problems with delegation.
  Margaret Wasserman: I'm going to argue with the wording of this point
(though it may have been my fault) I have had conversation, there is a
problem with this as a problem. It is solution, i.e. we need to state it as
a problem, with effective practices being the solution.
  Eric Hickman: I don't find problems with eng quality, it is that over time
that we apply ourselves less and less, especially to achieve consensus, and
to get it you alienate half of those involved. How to get more people
involved, less on fighting about driving.
  Elwyn: We can get hijacked if we are not careful
  Keith Moore: We are talking too much about tools and such, we don't
understand the problem that we are dealing with. The feedback loops now are
too long. Some of this stuff is really, really too basic.
 On 2.3 Lack of Engagement
  Keith Moore: Don't disagree, but think it needs to be restated as "how
much time people are committed to the process" and stop talking about the
good old days.
  Christian R: I had problems with explosion of information, 400 emails a
day for one SIP list.
  Randy Bush: There are a lot more people involved then we may think.
  Nathan L: There is a lot more money in using the standards then in making
them, which is part of the problem.
  Steve Trowbridge: I'm not concerned by spectators, but I am concerned with
those that are frightened, disengaged, or have no traction
  Eric Gutman: Rudeness and such causes people to leave mailing lists
  Harald: WG chairs have right to remove people, they have they power to do
so, and should.
  James K: I know of a case where physical violence
  Steve Trowbridge: I've seen a few cases where AD or WG have done so, but
in most wait far too long.
  Jonas H(?): Sometimes it isn't abuse, but sometimes is "you don't
understand" or otherwise intimidated and making people "look stupid"
prevents participation.
  Spenser D: The IETF has become kinder and gentler, maybe partly because we
are not at 30% new attendees. However, we could grow again and it could get
worse again. Where I am in the IETF I don't see a need for it today, but put
come back.
  Keith Moore: This document doesn't discuss how meetings work, and how
lists work, and we should have a section on each.
 On 2.4 Authority and Influence
  Elwyn: This is the part that has given me the most heartache
  Keith: Fundementally disagrees with this section, there is no problem with
this. It is clear that there is a perception of this, it is clear that a
perceptual differences are part of the problem, but I don't think it is
fact. Need more observations rather then conclusions.
  Robert Hinden: I agree with this section. It meets my perception of
reality.
  Steve Trowbridge: While there is nothing in the proceedures that causes
this to be the case, but I do think in practice that this is. There are
backroom deals that are set in stone that cause these perceptions. There is
a window prior to draft where insiders only get to be involved, otherwise
they get rejected. I think it is reality.
  Henning: Quantity has change a lot, but has stablized. However,
qualititivily it has change from WG level to AD level. Perception is that
anything of any signicance now needs AD involvement.
  Margaret Wass.: This section is where some of the scaling issues of the
IETF are showing up. I don't think it is the fault of anyone involved. We
ask too much, so no wonder they fail. We should change the jobs.
  Jonas Y(?): I agree with the previous speakers. No single person at fault,
don't fingerpoint. But it is a fact that people do the work in good faith,
but the process has problems. Keith's problems with perceptions vs. fact --
both are equally important. It is there.
  Elwyn: We need to talk about perception vs. fact more in next rev.
  Keith Moore: No one will disagree with perception that IESG hasn't scaled,
but qualifying it as 'authority' is inappropriate. I was in IESG 4 years,
you really can't block things. This isn't an accurate statement. It do agree
possible for an AD to be capricious, we should fix this.
  Marshall Rose: There are some former ADs that say that this has happened,
in spite of what Keith said.
  Keith D: We need decisions need to be more transparent -- they are now so
hidden.
  ?: Why does the IESG decide which WGs can be formed, it arbitrary and
isn't transparent.
  Eve S: I've taken a reprieve, so I have some perspective. It isn't that
more broken then it used to be (laugh from audience) but I do agree that
scale has changed. We need more outreach and recruiting, and move more
people up. There are lots of scalability problems.
  Margaret Wass.: Others problems besides scaling. to what extent and where
should progression decisions be made? How much in WG chair or AG. Also, is
our selection processes fair?
  Jonas Y(?): You shouldn't have to have been in the IESG to understand how
the IESG works. Currently that is not the case. This is a bad practice and
alienates people from the process. People don't know what is going on, "you
don't understand it because you haven't been there" isn't acceptable.
  Elwyn: Cultural issues also cause problems.
 On 2.5 Design Making Flaws
  No comments from floor!!!
 On 2.6 Training
  Christopher Allen: Medium issues are part of the problem. Email has
specific advantages
  Pete Resnick: (?)
  Margaret: WG chairs, and what they do needs more definition and training.
  Christopher Allen: Mentoring is another issue -- we need more explicity
have mentorship. Or even announce leadership changes 6 months in advance of
taking on the job, giving a transition.
  Harald: Paradox is "ruling class" is used here and wasn't contentious, but
it was in a previous slide
 New stuff
  Robert Hinden: There are problems that with getting new people, and
readying them to be come the new leaders. I have the feeling that it has
become more exclusive. There is a perception of cycling from IAB to IESG,
IESG to WG, etc.
  Joel H: We talk about authority to block things - I disagree with Keith
that this doesn't happen. But sometimes that is appropriate. (gave example
from 82) The AD said it doesn't belong, and forced a change. So removing the
authority might cause more problems then it causes.
  Robert Hinden: In the beginning problem, that is good, but if it is at the
end of the process, it is bad. This type of blocking needs to happen sooner.
  Elwyn. A symptom of feedback mechanisms of falling apart.
  Ted H: Part of the problem is the metaphor "Send up to the IESG" which
colors the perceptions. There are many things in the document that have a
series of metaphors of consensus, and we use the words of consensus, but in
fact, that sometimes we use disputation, but when we use disputation with
consensus can cause acrimony. Since our mechanism imply consensus, those
points where we must have decisions. We need metaphors to match our
recommendations.
  Dave Crocker: I'm struck how important and interesting the previous folks
that have been said. On the other hand, I think we have plenty of mechanism
to call the question, but I don't think we do it enough. We have done it
without acrimony often. It may not be a structural problem, but a management
problem. For instance we sometimes choose between two when we don't have to,
sometimes we don't choose when we should. The problem may not be a
structural problem but a training one. Which is cool because we can fix
this.
  Jonas Y: We have lost a lot of "Rough Consensus and Working Code". It used
to be that you'd decide a solution by fielding it. We need to stop the
"stopping" of ideas.
  Eve S: Re: training. We've mentioned specific documents that are good, but
we need more documents on being a WG chair.
  Ted H: We'd like to make decisions that are good for the internet as a
whole. We so narrowed the scope of WGs that we can't see the full picture.
We have narrowed them too much such that if we make the 'market decide' the
consequences could be quite serious.
 Elwyn: Very constructive discussion. Thank you for not ripping me from limb
to limb.

* AOB, milestone review, wrap-up (Chair)
 Please read draft!
 Please post on list!
 Read new process draft in a few weeks.
 We are serious about doing our milestones.
 To do that you have to participate.