Scribe from Problem-Statement WG Meeting, SF
Christopher Allen
ChristopherA at AlacrityManagement.com
Fri Mar 21 11:35:42 CET 2003
Problem-Statement working group meeting
* Blue sheets/Agenda Bashing
Agenda Review
* Status (chairs)
Active as of 6 March 2003
Chairs: Avri Doria <avri at acm.org>, Melinda Shore <mshore at cisco.com>
Editors & Editing Team (list of people)
Answer to floor question: Editor team is for both documents
* Charter Discussion (chairs)
The charter has not been controversial
The workgroup is to be more analytic then descriptive
The WG has a very aggressive deliverable schedule, ending in October
First document to be done in a couple of weeks
First version of a document is a proposal, not a final
We will review at IESG plenaries in future meetings
Christopher Allen: Would like to add to charter "...and we should have a
clear understanding of existing procedures and rules which are effective, so
that we may preserve them."
* Document process performance (Henning Schulzrinne)
IETF Goodput Measurements: Packets Size, Delay & Throughput
For our deliverables
Determine throuput, packetsize, delay in publication of RFCs as a function
of time
Delay measured from --00 to RFC, by title
Based on:
rfcindex.txt, IMR, IETF announce
Inserted into EDAS/netbib (sql database)
Data is very poor, very difficult to compute
Data based on matching RFC title with I-D title
Only IETF announcement list contains original I-D filename!
This information is lost! No archive earlier then 1998!
We have lost our announcement history older then 5 years.
Lack of this old information is causes some data analysis problems
RFC Bandwidth and packet size (chart)
RFCs published per year
50 or so in 60s
100 or so in 70s
begins to flatten and jump
150-200 or so per yearin 80s, 90s, and 00s
RFC avg. pages
Growing pages, but not as much as Henning thought
I-D Bandwidth
Though RFCs production is flat, I-Ds are growing fast
500 in early 90s, 2000 in mid 90s, 3500 in 2001
Is not significantly increasing since 2001
Substantiated by other numbers like attendance
This is not just 00 IDs, but all i-d actions changes are noted
Hasn't done an analysis of only 00 drafts
RFC Delay
Delays in early 90s 5 months, mid-90s 15 months
since 97 has been roughly 20-25 months
RFC Delay Histogram
There is a clustering at 2 years delay, but there are some significant
tails longer then 2, with 5 years.
Lots of comments from floor about definitions, questions about title
matching, questions of whether all was captured, hopeless of capturing some
info.
Comment of floor: frequency of updates is a problem
Caveats from data
Noisy Data
Lots of ways of formating I-D announcements
Lots of editing areas
IMR is incomplete
Some RFCs were even never announced!
The estimate is likely low due to I-D "elevation"
- draft-alice-problem -> draft-ietf-problem
- drafts merged
(lost some stuff from slide, speaking running out of time)
Suggestions
Sampling instead of complete statistics
Could get delay from final I-D to RFC with some effort
Track biography of random RCS and try to ascertain delay compenents
- one I-D iteration per IETF
- waiting for other documents
long delay waiting for author fixes in RFC editor queue
IESG "Tabling"
Dont' care about minor statistical variation
(lost one item)
Summary
Statistics reasonable stable since ~1988
- Avg. RFC size has not increased since 1990,
(lost last sumary item due to rush)
* Issues in IESG Review (Eric Rescorla)
("something of in the water of a desire to do statistical research", wishes
he had Henning's data)
Analysis of 4+ years of IESG Review
IESG Review is an important process
(lost points due to speed)
Basic question: How long do things take from last call to approval
mean 147 days
median 93
standard dev 149 days
long tail of 600 days
Drafts that are approved as-is (no changes) are faster, but not much
average is still 78
mean is 54(? not sure if scribed correctly)
The areas differ quite a bit in speed
INT, O&M, and TSV are significantly faster
Applications and Security are slower
Rate of approval is about twice as fast from slowest to fastest
Big diferencs by AD in output volume
Document Issues
Churning consumes time
each revision consumes 58 +/- 6 days
lots of noice (r-squared=.39)
But the volume of changes doesn't make any difference
once you control for number of revisions
This is rather surprising
Length of document doesn't matter
This is also surprising
Things aren't getting and slower, but not faster
Comments from floor
Dave Crocker: There are some problems in methodology, some problems with
normal, massaging data and using standard deviation can cause problems. I'm
not saying the math was wrong, but use of those statistics in that
environment is risky, because the community of professionals use normal
data, but don't know how to process that data. There is a difference between
the math world and the social research world.
Eric Resc.: Disagrees
Bill Sommerfield: As a WG chair, I'm frustrating, I get one bug report, we
fix it, submit, get one more bug report, fix it, etc. Do you have data on
small changes
Eric Resc.: From 0 to 4 review cycles typically
Aaron Falk: One of the things that gets in the way of understanding that
is because the AD review, other undocumented aspects
Eric Resc.: One difference between AD groups may because exactly this.
* IETF process and architectural input (James Kempf)
Presentation title "Helping the Internet to Age Gracefully"
One of our problems is that the Internet is getting older
Heard around the IETF
"We don't do any <insert your favorite explitive> architecture"
Can't use the word architecture, we have to use framework
Heard around 3GPP
IETF does philosopy
Internet Architecture Historically
Architecture historically
Articulation of broad design principals
Development of conswquestions
(lost items to to speed)
What is the problem
Need some way to maintain consistency
New componenents must integrate with existing system
Removing or extending old components must honor interdependecies
sometimes interdependencies now cross WGs and even Areas
Catching them at IESG REview is often too late to affect the base design.
By the time the drafts get the IESG, they can't do much more then throw
back, or only get the worst problems fixed
The architecture the wrong word?
Maybe "System Design" (John Klensin)
Possible Consquences of Ignoring this problem
More drafts returnted to WG during IESG review due to conflicts
Occasional missed interdependencies turning up after draft has gone to PS
- Interoperability problems
- Delays in successful deployment
- Problems in products
- increase expense
Increased lack of transparency and understanding in Internet design
- One of the advantages that internet has over telephony world is that it
is relatively easy to understand, not baroque.
Comment from Robert Hinden(?): Proposed Standard means proposed standard,
it is the experience of those who risk implementation of proposed that
drives to draft standard. Something may cycle at PS several times before
going to DS.
Bill Sommerfeld: It is so difficult going to proposed standard, that most
people give up after draft standard is done. You are out of energy often
right before Proposed Standard, much less after to move to Draft Standard
Henning: We need more case studies. There are a lot of particulars lost.
James comments: agrees that we need more studies, is doing a draft
* Issues in working group structure (Ted Hardie)
Working Group Participation AKA -- The "Stuckee" problem
(comments about the amount of time lost making laptops display)
Intro
Has written on a draft on 'stuckee's
Belinda challenged Ted to present it as a "problem"
What are WG accountable for
Within charter:
make good engineering decisions
specify them
clearly
promptly
Working Group Theory
Historicaly are defined by mailing list, all decisions require consensus
oflist
No membership requirements
tech commetns fromanywhere, cross-fertilization is possible
specific open calls to all
Opening vs commitment issue
making a comment on a document does not imply that you are taking
responsibility for the work of the working group
That ambiguuity makes it difficult to predition
How much attention
estimate time
estimate effort
Openness can make it difficult to make anyone other then SG chair and
authors accountable.
How do we retain openness
We need commitment to
Make decisions
Produce docs
Review docs for accuracy, readability, and usefullness
without
shutting out the technical comments from those outside the WG or Area
Comments from Dave Crocker: Do you mean that the problem 'people consume
time and bandwidth and don't contribute." I'm not sure what the commitment
step buys. It is true that WGs vary over time, but over time they loose
people and change participation, but thus a loss of constintuancy. If they
don't show up, they don't get to speak: that is the test that makes things
work.
Reply by Ted: Not precisely the problem that I'm stating, it is more how
can WGs chairs can do both. This ambiguity causes it to be difficult for
estimates for WGs to estimate and predict progress on work, in particularly
as multiple groups had interpendencies. I believe that this needs to be as
much an open process as it can, but that openness is so unbounded over time
cause problems. We want commitments that survive over time, that if you
commit you should follow through.
Dave Crocker: Everything you said is extremely reasonably, but is very
different then the philosophy and history and vision of the IETF. It may be
that to fix it is to change the statement of the problem. Dave does realize
that since he has spoken up on this topic that he needs to work on it ;-)
Spencer Dawkins: Other groups (non-IETF groups) also have problems with
interdepencies with the IETF
Robert Hinden: Sometimes work that is required by another working stops
because the providing group looses energy, whereas the dependent group has
it.
* Process Document (Margaret Wasserman)
Title: Problem Resolution Process
Talking about the second deliverable of this group
Process Document
A proposal for a process to develop solution to the problems identified
Describes the process of the process
Milestones
Mar 03 First I-D of process document
Jus 03 Process review at IESG plenary
Aug 03 Process document submitted to IETF
I need comments on list on discussions already there
Document in progress
Thoughts about change in the IETF
High-level process considerin
Mechanisms, knobs, leves
Probem Perception
Good time to hide problems
Tendency to sit back and rest on laurels
Bad Times Hide Successes
Tendency to be over-critical and reactive
Solution
Consistent focus on operational improvement
Build a culture to do this
We need to do this in the IETF
Core Values (from IESG Plenary 2002)
Cares for the internet
technically competent
oepn process
volunteer core
"We reject kings, presidents and voting. We believe in rough consensus and
working code"
Not core values
The division into WGS and AReas
The three-step standards process
The ASCII format for RFCs and IDS
The format of IETF meeting
The way mailing lists works
Current Environment
Tension and Emotionalism
Us vs Them mentality
Frustration on all sides
Lack of trust between leaders/non-leaders
Commitment to improvement
People on all sides seem committed to improving the IETF
Moving Target
The IETF is a moving target
This make things tricky
Process of acknowledging/identifyin problems is already creating chagne
Can't ask the IETF to stand still while we measure it
Major Process Options
(I don't know where we stand on these tradeoffs)
Evolution or Revolution
Granular or Monolithic
Deliberate or Immediate
Grass Roots or Coordinated
Details
Evolution
Focus on iterative improvement
indentify
formulation
reassess
repeat
Most appropriate when things are working
Revolution
Focus on large scale change
Determine type and scope of change
Best when fundemental changes are needed
Nothing left to loose
Granularity
Address separate problems separately
several smaller efforts
different process choices
allow execution at different priorities
Works if separable problms
Mololithic
(lost, too fast)
Deliberat
Careful process with several steps
Best for non emergency probems
good for complex problems
need to measure what is good
Immediate
Determinte a change and execute
Base
Best in an emergency
avoid analysis paralysis of the deliberate approach
Grass Roots
Improvement efforts start from below
Let those who have an idea propose
leadership determines which have priority
Coordinated
Establish a time responsible to the problem
Problem space oriented
Available Mechanismes
Ask leadership to correct
Hold a BOF
Form a WG
Form a Design Team
Assign problems to individuals
Others?
Knobs and Levers
Update existing process documents
RFC 2026 etc.
Develop new processes and roles
IESG charter, proceedures
Organize independent effotts
Organizational development training
Where do we go from here?
Need to develop intitial draft in next few weeks
Should be responsive to the problems
What should it say?
Not much commetns on list.
James Kempf comment: People are assuming that there is one solution, and
that the entire IETF will adopt it. We should try a bunch of different
approaches and see what ones work. This means that we have to be willing to
accept that some will not work.
Margaret replies: That is one of the possible solutions above.
Harald: Paradoxical statement. Fixing our short term problems may be
harmful of the internet. My conclusions are that fundemental problems with
the IETF may cause serious problems of the internet. In five more years that
we may be dead if we don't make fundemental changes. That said, we have to
do some short term fixes! A paradox.
Question from Floor: Does that mean that fundemental change has broad
consensus?
Harold: <nods affirmatively>
Dave Crocker: What will be the experimental group? We have been doing an
experiment for 13 years, we do the best we can, and it works. We don't have
enough controls to unerstand. Effects can be ranged, varied, and variable.
Bob Hinden: Harald is right, we need to make fundemental changes.
Harald: How do we reach consensus, if we can't reach it, I will go home.
This is an open process, I need contributions.
Chairs: Keep talking about this on the list!
* Problem definitions/scoping document (Elywn Davies)
IETF Problem Statement (discussion of draft)
Root Cause Approach
differentiate derivitive problems from primary
current org of draft is provisional
Initial thoughts
The problems are not new
the problems are not IETF specific
many are consquense of growth
First Pass
750 emails and 3 drafts
Topics extracted and filed, attributed to first contribution
Categoried used baseic structure and process of the IETF
not retrofitted
Next Steps
Discussion of breakdown andetails
Today and continued onlist
looking for
alt causes
better breakdown
better wording to clarify this is not a witch hunt
In parallel - creation of Process Recommendation
(missing some items)
Initial List of Root causes
(list from draft)
Comment from floor: You should be commended for taking this on. HOwever,
Reading draft bothered me. It makes things sound like facts that are
perceptions. I don't feel that way for some of the pieces. That they were
stated as fact is a problem.
Comment from floor: We don't need to see this list, we saw it in plenary.
Lets not waste the 10 minutes on going over the list again.
Dave Crocker: I'm part of the editing team, I want to note that Elwyn
turned out that document on his own, and it isn't a thankless job, as we
think we've thanked him a lot. I don't care if it is written as facts, but
it does have value as it is because there is no such thing as 'fact'. We
need to get a rank ordering of consensus. One important issues is that we
need to make some priorities, some things we can do now with big impact,
etc. It is a big matrix of decisions.
Graham: If there are some people that have a perception of a problem, that
is a fact. Even if they are wrong, the perception is a fact.
(this scribe refers you to draft for the list of draft)
On 2.1: Mission
Christopher Allen: We are not looing at the culture, the mission can be
too dry. What drives people to help are the culture and values
On 2.2: Effective Engineering Practices
Robert Hinden: This maybe needs to be broken into two items, project
management and quality management.
Henning: Lack of granularity of milestones, lack of support, lack of
priorities. We need a 'contract' with the authors.
Elwyn. There are problems with delegation.
Margaret Wasserman: I'm going to argue with the wording of this point
(though it may have been my fault) I have had conversation, there is a
problem with this as a problem. It is solution, i.e. we need to state it as
a problem, with effective practices being the solution.
Eric Hickman: I don't find problems with eng quality, it is that over time
that we apply ourselves less and less, especially to achieve consensus, and
to get it you alienate half of those involved. How to get more people
involved, less on fighting about driving.
Elwyn: We can get hijacked if we are not careful
Keith Moore: We are talking too much about tools and such, we don't
understand the problem that we are dealing with. The feedback loops now are
too long. Some of this stuff is really, really too basic.
On 2.3 Lack of Engagement
Keith Moore: Don't disagree, but think it needs to be restated as "how
much time people are committed to the process" and stop talking about the
good old days.
Christian R: I had problems with explosion of information, 400 emails a
day for one SIP list.
Randy Bush: There are a lot more people involved then we may think.
Nathan L: There is a lot more money in using the standards then in making
them, which is part of the problem.
Steve Trowbridge: I'm not concerned by spectators, but I am concerned with
those that are frightened, disengaged, or have no traction
Eric Gutman: Rudeness and such causes people to leave mailing lists
Harald: WG chairs have right to remove people, they have they power to do
so, and should.
James K: I know of a case where physical violence
Steve Trowbridge: I've seen a few cases where AD or WG have done so, but
in most wait far too long.
Jonas H(?): Sometimes it isn't abuse, but sometimes is "you don't
understand" or otherwise intimidated and making people "look stupid"
prevents participation.
Spenser D: The IETF has become kinder and gentler, maybe partly because we
are not at 30% new attendees. However, we could grow again and it could get
worse again. Where I am in the IETF I don't see a need for it today, but put
come back.
Keith Moore: This document doesn't discuss how meetings work, and how
lists work, and we should have a section on each.
On 2.4 Authority and Influence
Elwyn: This is the part that has given me the most heartache
Keith: Fundementally disagrees with this section, there is no problem with
this. It is clear that there is a perception of this, it is clear that a
perceptual differences are part of the problem, but I don't think it is
fact. Need more observations rather then conclusions.
Robert Hinden: I agree with this section. It meets my perception of
reality.
Steve Trowbridge: While there is nothing in the proceedures that causes
this to be the case, but I do think in practice that this is. There are
backroom deals that are set in stone that cause these perceptions. There is
a window prior to draft where insiders only get to be involved, otherwise
they get rejected. I think it is reality.
Henning: Quantity has change a lot, but has stablized. However,
qualititivily it has change from WG level to AD level. Perception is that
anything of any signicance now needs AD involvement.
Margaret Wass.: This section is where some of the scaling issues of the
IETF are showing up. I don't think it is the fault of anyone involved. We
ask too much, so no wonder they fail. We should change the jobs.
Jonas Y(?): I agree with the previous speakers. No single person at fault,
don't fingerpoint. But it is a fact that people do the work in good faith,
but the process has problems. Keith's problems with perceptions vs. fact --
both are equally important. It is there.
Elwyn: We need to talk about perception vs. fact more in next rev.
Keith Moore: No one will disagree with perception that IESG hasn't scaled,
but qualifying it as 'authority' is inappropriate. I was in IESG 4 years,
you really can't block things. This isn't an accurate statement. It do agree
possible for an AD to be capricious, we should fix this.
Marshall Rose: There are some former ADs that say that this has happened,
in spite of what Keith said.
Keith D: We need decisions need to be more transparent -- they are now so
hidden.
?: Why does the IESG decide which WGs can be formed, it arbitrary and
isn't transparent.
Eve S: I've taken a reprieve, so I have some perspective. It isn't that
more broken then it used to be (laugh from audience) but I do agree that
scale has changed. We need more outreach and recruiting, and move more
people up. There are lots of scalability problems.
Margaret Wass.: Others problems besides scaling. to what extent and where
should progression decisions be made? How much in WG chair or AG. Also, is
our selection processes fair?
Jonas Y(?): You shouldn't have to have been in the IESG to understand how
the IESG works. Currently that is not the case. This is a bad practice and
alienates people from the process. People don't know what is going on, "you
don't understand it because you haven't been there" isn't acceptable.
Elwyn: Cultural issues also cause problems.
On 2.5 Design Making Flaws
No comments from floor!!!
On 2.6 Training
Christopher Allen: Medium issues are part of the problem. Email has
specific advantages
Pete Resnick: (?)
Margaret: WG chairs, and what they do needs more definition and training.
Christopher Allen: Mentoring is another issue -- we need more explicity
have mentorship. Or even announce leadership changes 6 months in advance of
taking on the job, giving a transition.
Harald: Paradox is "ruling class" is used here and wasn't contentious, but
it was in a previous slide
New stuff
Robert Hinden: There are problems that with getting new people, and
readying them to be come the new leaders. I have the feeling that it has
become more exclusive. There is a perception of cycling from IAB to IESG,
IESG to WG, etc.
Joel H: We talk about authority to block things - I disagree with Keith
that this doesn't happen. But sometimes that is appropriate. (gave example
from 82) The AD said it doesn't belong, and forced a change. So removing the
authority might cause more problems then it causes.
Robert Hinden: In the beginning problem, that is good, but if it is at the
end of the process, it is bad. This type of blocking needs to happen sooner.
Elwyn. A symptom of feedback mechanisms of falling apart.
Ted H: Part of the problem is the metaphor "Send up to the IESG" which
colors the perceptions. There are many things in the document that have a
series of metaphors of consensus, and we use the words of consensus, but in
fact, that sometimes we use disputation, but when we use disputation with
consensus can cause acrimony. Since our mechanism imply consensus, those
points where we must have decisions. We need metaphors to match our
recommendations.
Dave Crocker: I'm struck how important and interesting the previous folks
that have been said. On the other hand, I think we have plenty of mechanism
to call the question, but I don't think we do it enough. We have done it
without acrimony often. It may not be a structural problem, but a management
problem. For instance we sometimes choose between two when we don't have to,
sometimes we don't choose when we should. The problem may not be a
structural problem but a training one. Which is cool because we can fix
this.
Jonas Y: We have lost a lot of "Rough Consensus and Working Code". It used
to be that you'd decide a solution by fielding it. We need to stop the
"stopping" of ideas.
Eve S: Re: training. We've mentioned specific documents that are good, but
we need more documents on being a WG chair.
Ted H: We'd like to make decisions that are good for the internet as a
whole. We so narrowed the scope of WGs that we can't see the full picture.
We have narrowed them too much such that if we make the 'market decide' the
consequences could be quite serious.
Elwyn: Very constructive discussion. Thank you for not ripping me from limb
to limb.
* AOB, milestone review, wrap-up (Chair)
Please read draft!
Please post on list!
Read new process draft in a few weeks.
We are serious about doing our milestones.
To do that you have to participate.
More information about the Problem-statement
mailing list