interoperability testing

[Erik van der Poel]

At Google, we have been working on a tool and set of test cases in
HTML that allow us to test the browsers. (We haven't spent time on IM,
email, databases, etc yet.) We don't use CSS in these test cases, so
the results don't show any CSS-dependent differences. Basically, we
start WireShark (previously known as Ethereal) and then we capture
packets sent from browsers when we load the HTML test files in them.
Then we save the captured file and generate reports that show
differences between, say, one browser on different operating systems,
or different browsers on the same operating system.

For example, we can now clearly see the difference between Firefox 2
and 3 in the handling of non-ASCII text in the path part of the URL.
(Firefox 2 sends it in the original encoding of the HTML, while
Firefox 3 sends it in UTF-8.) We also test every ASCII value (0..127)
in every part of the URL. We look at both DNS and HTTP packets.

The results are not quite ready for publication. We will probably send
them to the browser developers first, and then to spec writers and
other interested folks.

Erik

On Sat, Jul 11, 2009 at 10:32 AM, Eric
Brunner-Williams<ebw at abenaki.wabanaki.net> wrote:
> I'm not sure there is a sensible single definition of IDNAbis
> interoperation by multiple implementations. I'm interested if one exists
> for IDNA2003.
>
> If one were to automate a test for two or more instances of a browser,
> possibly different snapshots of the same implementation, having common
> specification, variation in css handling between the two or more
> versions could result in "difference determined by test".
>
> We are fairly safe when we ask if the resolution narrative is
> indifferent to which of {bind8, bind9, ...} instances of and test
> configurations of caching and authoritative resolvers.
>
> We are also fairly safe when we ask if the call and return narrative of
> API use is portable within some framework of source code portability, if
> APIs are within the scope of interoperability and multiple independent
> implementations.
>
> How do the {IM,email,browser,database,...} application implementors
> propose to define meaningful functional interoperation between any two
> or more independent implementations of their applications, or across any
> test configuration of similar or dissimilar applications?
>
> Eric