[R-C] Strawman: Requirements

[Varun Singh]

Hi,

Comments inline,

[snip!]

>> Is there a proposal for what BW should be? We should test for
>> different values of BW.
>
> That's why I made it a free variable. I'm thinking that 2 MB and 200 KB need
> to be tested.

ok.

>>
>> What about delay? should it be variable or fixed one-way delay
>> (end-to-end delay)?
>>

Any comments on end-to-end delay: fixed 50ms? 100ms? variable 20-70ms?
i.e a few ms on 100 Mbit links and 10s of ms on the bottleneck link (BW).

>> Simulate only uni-directional media flows or B sends media to A and A' ?
>
> I suggest only uni-directional for the test cases - keep things as simple as
> possible.
> If we have a proposal that passes the tests for unidirectional, but fails to
> work for bidirectional, we should add a bidirectional test to the test
> suite.

Okay.

>>>
>>> All measurements are taken by starting the flows, letting the mechanisms
>>> adapt for 60 seconds, and then measuring bandwidth use over a 10-second
>>> interval.
>>>
>> I think initially running simulations for 60 seconds might be alright
>> but for measuring impact (to and from cross-traffic), probably running
>> longer simulations of 300s (~5 mins) might be useful?
>
> Yes - I would suggest doing it this way first, but once the test framework
> is built, also run more tests, and add stuff to the requirements test suite
> when it becomes clear that there are systems that pass the tests, but still
> fail to give adequate service.
>>>
>>> This seems like the simplest reasonable system. We might have to specify
>>> the
>>> buffer sizes on the border routers to get reproducible results.
>>>
>> For a simplest system this seems alright, however, there is a need of
>> verifying if the media sender adapts when other flows (TCP or other
>> media senders) start their sessions later or before the RTCWEB client.
>
> True.
>>>
>>> PROTOCOL FUNCTIONS ASSUMED AVAILABLE
>>>
>>> The sender and recipient have a connection mediated via ICE, protected
>>> with
>>> SRTP.
>>> The following functions need to be available:
>>>
>>> - A "timestamp" option for RTP packets as they leave the sender
>>
>> A new header extension can be defined for this, if the one in TFRC is
>> not useful.
>>
>>> - A "bandwidth budget" RTCP message, signalling the total budget for
>>> packets
>>> from A to B
>>
>> Is this like TMMBN (Notification) or should the direction be B to A?
>
> I was thinking receiver-side estimation and a TMMBR-like message that
> applies to the whole media flow.
> In internal work, we have used the name REMB - Receiver Estimated Media
> Bandwidth.

The reason I asked, was because the direction of the message is noted
as "from A to B" and TMMBR should flow "from B to A" and I wasn't sure
if it was TMMBR or was the sender telling the receiver what it was
using as the current bit rate. However, REMB clarifies it.

>>>
>>> REQUIRED RESULTS
>>>
>>> In each measurement scenario below, the video signal must be
>>> reproducible,
>>> with at most 2% measured packet loss. (this is my proxy for "bandwidth
>>> must
>>> have adapted").
>>>
>> This seems like a reasonable upper-bound. However, the performance may
>> vary depending on wireless/wired environment and the amount of
>> cross-traffic.
>
> Yes. Again - if we test in that environment and detect that it makes a
> difference....
>>>
>>> If A sends one video stream with max peak bandwidth<  BW, and there is no
>>> other traffic, A will send at the maximum rate for the video stream. ("we
>>> must not trip ourselves up").
>>>
>>> If A and A' each send two video streams with MPB>  1/2 BW, the traffic
>>> measured will be at least 30% of BW for each video stream. ("we must play
>>> well with ourselves")
>>
>> Just to clarify, for the whole simulation the average media rate for
>> each should be over 30% of BW?
>
> For the ten seconds of "measurement time".

is this a moving average of 10s or 10s intervals? because I am not
sure why 10s instead of per second?

>>>
>>> If A sends a video stream with MPB>  BW, and there is a TCP bulk transfer
>>> stream from LAN X to LAN Y, the TCP bulk transfer stream must get at
>>> least
>>> 30% of BW. ("be fair to TCP")
>>>
>>> If A sends a video stream with MPB>  BW, and there is a TCP bulk transfer
>>> stream from LAN X to LAN Y, the video stream must get at least 30% of BW.
>>> ("don't let yourself be squeezed out")
>>>
>>> Mark - this is almost completely off the top of my head. I believe that
>>> failing any of these things will probably mean that we will have a system
>>> that is hard to deploy in practice, because quality will just not be good
>>> enough, or the harm to others will be too great.
>>>
>> I'm wondering if 30% is a magic number? :)
>
> I pulled it out of a hat :-)
> I considered 25 and 40, and decided that 30 "felt" better. Very scientific!
> - but it's always better to name a number than to not name a number, because
> then people have to say if they want to go lower or higher. I've been in
> discussions where people talked about "low loss", and it turned out one was
> talking about 1% loss, and the other was talking about 0.001% loss.......

Yes relative terms can mislead. For these simple simulations this
might be an okay metric to use but I am not sure if this will be true
for more complex use-cases (with more competitions or different types
of cross traffic).

>>>
>>> There are many ways of getting into trouble that won't be detected in the
>>> simple setups below, but we may want to leave those aside until we've
>>> shown
>>> that these "deceptively simple" cases can be made to work.
>>>
>>> We might think of this as "test driven development" - first define a
>>> failure
>>> case we can measure, then test if we're failing in that particular way -
>>> if
>>> no; proceed; if yes; back to the drawing board.
>>>
>>> What do you think?
>>
>> Without running a simulation, it is hard to say if this is reasonable.
>
> If in either of these scenarios, we can argue that we have a well
> functioning system and the test still fails, we should make the test
> parameters less stringent.

Sure, we can discuss this when we come to it.

-- 
http://www.netlab.tkk.fi/~varun/