X.400 products are conformance and interoperability tested
Most X.400 products claim to conform to some profile - ENV 41021, NIST
Stable Agreements and so on and so forth.
This is IMHO worth considerably less than is apparent at the
surface. Consider:
- Some vendors claim conformance without having tested it with an
independent tester. Either because they feel that their inhouse
test is good enough, or because they don't see that the price
for external testing is worth the certificate.
- External conformance tests are expensive, which means that you
run them only a few times. Usually, you do NOT run them for
each release, patch version or product.
- Conformance tests only test what conformance tests test. That
means, for instance, that if a certain bug is only provoked by
input generated from a LAN Email system, or from the product's
SMTP gateway, there's no chance that this will be detected by
a pure X.400 conformance test.
All that said, the conformance tests are probably one reason why the
community of X.400 implementors pay considerably more attention to
what the standards actually say than Internet mail implementors do; if
you're told at the outset that you have to pass a conformance test at
the end, you think in terms of passing the test when designing the
system.
Most Internet mail products are interoperability tested; they're
connected to the Internet, and get to send and receive some mail
before they are shipped.
This is a considerably less stringent test, because:
- Most E-mail systems on the Internet are liberal in what they
accept; they will accept lots of egregious standards violations
in incoming mail and "attempt to do the best they can".
- Testing will almost always involve only a few out of the
thousands of different E-mail systems out there. Most of these
will not stress the limits of what's permitted by the
standards, but for every limit, there are some that will.
- Lots of current E-mail systems break standards. That influences
both the code shipped and the mindset of the developers; it
cons them into believing that "standards are not important".
The last point is important. Some products, for example early releases
of Microsoft's and Lotus' SMTP gateways, were incredibly badly
designed, full of bugs and totally out of conformance with the
standards. They seemed to have been written with a mindset of "slap
something together that works part of the time; nobody cares about
standards on the Internet anyway".
(The developers were justly flamed by the Internet community, and later
releases have fixed a lot of the bugs; the attitude of the Internet
community may be the Internet's only answer to conformance testing...)
Harald.T.Alvestrand@uninett.no
Last modified: Tue Oct 8 08:51:03 1996