[R-C] Strawman: Requirements

[Harald Alvestrand]

On 10/19/11 12:26, Varun Singh wrote:
> Hi,
>
> Comments inline,
>
> [snip!]
>
>>> Is there a proposal for what BW should be? We should test for
>>> different values of BW.
>> That's why I made it a free variable. I'm thinking that 2 MB and 200 KB need
>> to be tested.
> ok.
>
>>> What about delay? should it be variable or fixed one-way delay
>>> (end-to-end delay)?
>>>
> Any comments on end-to-end delay: fixed 50ms? 100ms? variable 20-70ms?
> i.e a few ms on 100 Mbit links and 10s of ms on the bottleneck link (BW).
I'd think that tests at 5 ms and 100 ms delay on the "narrow" link 
should give us reasonable scenarios - one simulates adjacent houses, the 
other one simulates the other side of a continent.
>>> Simulate only uni-directional media flows or B sends media to A and A' ?
>> I suggest only uni-directional for the test cases - keep things as simple as
>> possible.
>> If we have a proposal that passes the tests for unidirectional, but fails to
>> work for bidirectional, we should add a bidirectional test to the test
>> suite.
> Okay.
>
>>>> All measurements are taken by starting the flows, letting the mechanisms
>>>> adapt for 60 seconds, and then measuring bandwidth use over a 10-second
>>>> interval.
>>>>
>>> I think initially running simulations for 60 seconds might be alright
>>> but for measuring impact (to and from cross-traffic), probably running
>>> longer simulations of 300s (~5 mins) might be useful?
>> Yes - I would suggest doing it this way first, but once the test framework
>> is built, also run more tests, and add stuff to the requirements test suite
>> when it becomes clear that there are systems that pass the tests, but still
>> fail to give adequate service.
>>>> This seems like the simplest reasonable system. We might have to specify
>>>> the
>>>> buffer sizes on the border routers to get reproducible results.
>>>>
>>> For a simplest system this seems alright, however, there is a need of
>>> verifying if the media sender adapts when other flows (TCP or other
>>> media senders) start their sessions later or before the RTCWEB client.
>> True.
>>>> PROTOCOL FUNCTIONS ASSUMED AVAILABLE
>>>>
>>>> The sender and recipient have a connection mediated via ICE, protected
>>>> with
>>>> SRTP.
>>>> The following functions need to be available:
>>>>
>>>> - A "timestamp" option for RTP packets as they leave the sender
>>> A new header extension can be defined for this, if the one in TFRC is
>>> not useful.
>>>
>>>> - A "bandwidth budget" RTCP message, signalling the total budget for
>>>> packets
>>>> from A to B
>>> Is this like TMMBN (Notification) or should the direction be B to A?
>> I was thinking receiver-side estimation and a TMMBR-like message that
>> applies to the whole media flow.
>> In internal work, we have used the name REMB - Receiver Estimated Media
>> Bandwidth.
> The reason I asked, was because the direction of the message is noted
> as "from A to B" and TMMBR should flow "from B to A" and I wasn't sure
> if it was TMMBR or was the sender telling the receiver what it was
> using as the current bit rate. However, REMB clarifies it.

Yup.
Branching into design for a moment: With sender-side bandwidth 
estimation, I am worried about packet rates; since RTCP packets don't 
piggyback on return traffic like TCP ACK does, having a sender-side 
computation based on RTCP (such as the basic TCPFR) requires that you 
send RTCP packets very frequently - could be up to one RTCP for every 
RTP packet in short-RTT scenarios.

Computing bandwidth at the receiver and sending the computed rate to the 
sender allows you to react quickly to changes AND keep a low RTCP 
frequency when you don't have a congestion problem. This of course 
requires that you have a sender-side algorithm that reacts appropriately 
when congestion becomes so bad that most of the RTCP packets get 
lost..... always tradeoffs!

>>>> REQUIRED RESULTS
>>>>
>>>> In each measurement scenario below, the video signal must be
>>>> reproducible,
>>>> with at most 2% measured packet loss. (this is my proxy for "bandwidth
>>>> must
>>>> have adapted").
>>>>
>>> This seems like a reasonable upper-bound. However, the performance may
>>> vary depending on wireless/wired environment and the amount of
>>> cross-traffic.
>> Yes. Again - if we test in that environment and detect that it makes a
>> difference....
>>>> If A sends one video stream with max peak bandwidth<    BW, and there is no
>>>> other traffic, A will send at the maximum rate for the video stream. ("we
>>>> must not trip ourselves up").
>>>>
>>>> If A and A' each send two video streams with MPB>    1/2 BW, the traffic
>>>> measured will be at least 30% of BW for each video stream. ("we must play
>>>> well with ourselves")
>>> Just to clarify, for the whole simulation the average media rate for
>>> each should be over 30% of BW?
>> For the ten seconds of "measurement time".
> is this a moving average of 10s or 10s intervals? because I am not
> sure why 10s instead of per second?
I was thinking in terms of measuring the number of bytes delivered from 
second #60 to second #70 in the (simulation / test), and calling that 
"the measured value".
If we get significantly different numbers in second #70 to #80, that's 
worthy of investigation.

My reason for picking 10s is that I've done tests where it turned out 
that I was measuring effectively 4 events in the interval, and with a 
requirement that > 80% succeeded, a single packet loss was enough to 
break the test; if we measure over single seconds for a pass/fail 
criterion, I think a test will be a bit too sensitive to random 
combinations of events (double packet losses, for instance) for the kind 
of timescales we're talking about - there are only ~50 packets in the 
typical second for voice, and might be even fewer for scaled-down video 
(congestion-caused reduced-rate video may go down to 5 fps in some 
products). But it's another pragmatic number.
>>>> If A sends a video stream with MPB>    BW, and there is a TCP bulk transfer
>>>> stream from LAN X to LAN Y, the TCP bulk transfer stream must get at
>>>> least
>>>> 30% of BW. ("be fair to TCP")
>>>>
>>>> If A sends a video stream with MPB>    BW, and there is a TCP bulk transfer
>>>> stream from LAN X to LAN Y, the video stream must get at least 30% of BW.
>>>> ("don't let yourself be squeezed out")
>>>>
>>>> Mark - this is almost completely off the top of my head. I believe that
>>>> failing any of these things will probably mean that we will have a system
>>>> that is hard to deploy in practice, because quality will just not be good
>>>> enough, or the harm to others will be too great.
>>>>
>>> I'm wondering if 30% is a magic number? :)
>> I pulled it out of a hat :-)
>> I considered 25 and 40, and decided that 30 "felt" better. Very scientific!
>> - but it's always better to name a number than to not name a number, because
>> then people have to say if they want to go lower or higher. I've been in
>> discussions where people talked about "low loss", and it turned out one was
>> talking about 1% loss, and the other was talking about 0.001% loss.......
> Yes relative terms can mislead. For these simple simulations this
> might be an okay metric to use but I am not sure if this will be true
> for more complex use-cases (with more competitions or different types
> of cross traffic).
I think we'll have to be hard-nosed; if a test is so complex that we 
can't figure out whether it succeeded or not, we don't get very much 
value out of running it. The number will have to be adapted to the 
scenario (ie in a 5-stream flow I'd expect to perhaps use 10% of BW).
>>>> There are many ways of getting into trouble that won't be detected in the
>>>> simple setups below, but we may want to leave those aside until we've
>>>> shown
>>>> that these "deceptively simple" cases can be made to work.
>>>>
>>>> We might think of this as "test driven development" - first define a
>>>> failure
>>>> case we can measure, then test if we're failing in that particular way -
>>>> if
>>>> no; proceed; if yes; back to the drawing board.
>>>>
>>>> What do you think?
>>> Without running a simulation, it is hard to say if this is reasonable.
>> If in either of these scenarios, we can argue that we have a well
>> functioning system and the test still fails, we should make the test
>> parameters less stringent.
> Sure, we can discuss this when we come to it.
>