DRAFT Status of Work on IDNA2008 + IDNAv2

[Mark Davis]

Here are comments on the status. (I tried to update to the later doc, but
because it was only distributed in pdf, I had to do it manually, so I may
have missed something.)

Mark


On Fri, Mar 20, 2009 at 06:12, Vint Cerf <vint at google.com> wrote:

> DRAFT Status of work on IDNA2008
>
> 3/21/2009 0523 PDT
>
>
> Vint Cerf
>
>
> This brief summary is intended to provide some focus for the IDNABIS WG
> meetings
> scheduled for Monday and Tuesday, March 23 (1740-1940) and March 24
> (0900-1130).
>
> One goal is to try to assess rough consensus about the present
> documentation on the
> presumption that we are abiding by the ground-rules set forth in the
> charter of the WG.
> Another is to assess what the implications are for users, registries,
> registrars if
> IDNA2008 is adopted as it presently stands.  A third goal is to examine the
> implications
> of the IDNAV2 proposal from Paul Hoffman and contrast with adoption of
> IDNA2008.
>
> I fully recognize that consensus has to be assessed from mailing list
> exchanges, not
> merely from appearances at our face to face meetings.
>
> The material presented below is by no means intended to be more than a
> basis for
> discussion, and is not intended as a penultimate recommendation.


I think it would also be useful to mark where each is different from
IDNA2003.

>
>
> Background
>
>
>
> Under the IDNABIS charter, the IDNA2008 design as it now stands makes
> several
> specific assumptions or makes specific propositions to achieve a number of
> goals:
>
> 0. Avoid dependence on any specific version of Unicode through the use of
> rules
>    for determining PVALID characters based on Unicode character properties


add: "as much as possible". Exceptions may be necessary in some cases (and
are included in the draft tables).


> 1. No change to the deployed DNS server functionality (domain name labels
> limited to
>    ASCII and case-insensitive matching only)

[no change from IDNA2003]

>
> 2. Esszet, Final Sigma, ZWJ and ZWNJ, geresh and gershayim are PVALID
> characters
>     some of which are treated through contextual rules (there is still
> ongoing discussion
>     about the implications of these choices)


This is also a current feature of the drafts, but not required by the
charter. It is unclear whether this is actually consistent with the charter
or not. "This work is intended to specify an improved means to produce and
use stable and unambiguous IDN identifiers." Effectively, any IDN with the
first four characters is ambiguous between versions of IDNA in that it will
lead to different addresses.


> 3. Unassigned Unicode characters will not be looked up


Just a comment (no change): IDNA2003 had the slightly different goal:
unassigned Unicode characters will not be returned from the DNS.


> 4. No mapping of characters at least within the protocol specification


> 5. No modification of or dependence on Nameprep  (and thus no impact
>   on other protocols relying on Nameprep or Stringprep.)


> 6. Clear specification of valid "dot" form in a way that is consistent with
> DNS
>     protocol requirements.


IDNA2003 specified the dot form in a way that is consistent with DNS; that
is, it required no change of the DNS protocol, so this is no change. That
is, once in the ACE form, dots are dots.


> 7. Symmetry between native-character ("Unicode") and ACE ("Punycode")
>     forms of a label.


This may be a goal, but it is not achieved by the current drafts. There is a
strong asymmetry between them in that in lookup, an implementation need not
check that what appears to be an A-Label is one, but it must check that a
U-Labels is one (mostly). (Comment: I believe that this should be a goal: if
it is important to check those requirements, then it is important to test
both A and U Labels; if it is not important to test them, then it should not
be a requirement for either one.)


> 8. Conversion to an inclusion list of PVALID characters (as distinct from
> the
>    IDNA2003 posture that excluded only a few Unicode characters)


> 9. Improved terminology to make categories and types of labels more clear.
>    (Definitions)


> 10. Provide explanation for decisions and their motivations (Rationale) to
>     aid implementors, registries, registrants and users in understanding
> IDNA.


Rationale doesn't really provide explanation for motivations in enough
detail to be useful. I'd recast this as: "Provide informative background
material (Rationale) to aid ..."


> 11. Separately describe registration and lookup procedures to improve
> clarity


The goal is good, but the current drafts don't meet the goal. Whether it
increases clarity or not is unclear, since by doing so makes it difficult to
determine what the similarities and differences are between the two
processes. So drop "to improve clarity". (A relatively small recasting of
the text to make it precisely parallel between them (including numbering),
and point out precisely where the differences are, would meet this goal.


> 12. Specify tests to be applied at lookup time in an attempt to limit abuse
> of
>       IDNA at all levels of registration


That is not a change from IDNA2003. The tests are different, and are
expanded, but it is a quantitative difference, not qualitative. For example,
IDNA2003 did test bidi; we just think the IDNA2008 tests are better. And the
"in an attempt to limit abuse" is not true; the changes in IDNA2008 will
have a trifling effect on abuse at the very best, and introduce significant
opportunities for spoofing because of the 4 ambiguous characters. And
affecting the "phishing" problem is not a requirement of the charter. So
this item should be removed.


>
> 13. Clarify what is expected of IDNA-aware applications and domain name
>       "slots" with regard to invalid labels and future extensibility


These are still not nailed down in the current drafts. My expectations are
that once a domain name is valid, it remain valid for all time -- that is,
we are doing a one-time massive compatibility change, but there will be no
more changes that would affect compatibility. However, that is not captured
in the text, despite the charter requirement "This work is intended to
specify an improved means to produce and use stable and unambiguous IDN
identifiers."

Another major change is the introduction of a mechanism for changing IDNAs
on the fly via the context mechanism, with and associated process.


>
>
> Chartering and Re-Chartering
>
> (1) A Re-charter is needed if we abandon a significant fraction of the
> IDNA2008 goals
> and methods. IDNAv2, as described by Paul Hoffman requires a re-charter.
>
> (2) A Re-charter is needed if the WG decides to introduce mappings into the
> IDNA2008
> specifications since the basic assumption in IDNA2008 was that mapping
> would not
> be part of the specification.
>
> (3) It is possible that re-charter might not be needed if IDNA2008 adopts
> some
> IDNA2003 operations under a restricted set of conditions and only at lookup
> time for purposes of easing the transition to IDNA2008. This would be up to
> the
> AD and IESG presumably to decide.
>
> Basics for IDNA2003 and IDNA2008
>
> Both of these specifications use the Punycode algorithm to generate what
> IDNA2008 would call an A-label (ie. "xn-- <LDH compliant string>") from


Better expressed as an XN label. That terminology can be applied to both,
while A-Label only makes sense for IDNA2008.

>
> labels expressed as a string of characters drawn from a subset of Unicode
> defined characters.
>
> DNS matching is done in the servers by comparing the query string to the
> registered string in a case-independent fashion.  For IDNs, these
> comparisons
> are done after conversion into the "xn--" prefix form. For IDNs the case
> insensitive
> matching of the DNS servers applies only to the A-label form and not to the
> Unicode form. This means that the case-insensitive matching behavior of
> in traditional ASCII labels is not conferred on IDNs in their Unicode form.
>
> The case-insensitive comparisons between traditional LDH domain names is
> approximated under IDNA2003 by using CaseFold as a mapping guide on the
> Unicode strings being looked up. In addition, IDNA2003 also maps the
> so-called
> "compatibility" characters of Unicode into their counterparts. The same
> actions


=> "compatibility decomposable" characters of Unicode into their
counterparts
[Not all compatibility characters are decomposable and vice versa.]


>
> precede the registration of new domain names under IDNA2003.
>
> Unicode CaseFold maps to upper case and then map back to lower case.


This is not quite accurate; better would be to say "Unicode CaseFold maps
characters to lowercase values based on an an equivalence class formed by
including lowercase, uppercase, and titlecase mappings."


> Prior to Unicode 5.0, Ezsett became "SS" because there was no upper
> case, then became "ss" in the lower case mapping.  Under Unicode 5.0
> CaseFold was unchanged for  stability reasons. Consequently
> CaseFold (ESSZETT) is "ss" rather than lower case esszett even after
> the introduction of upper case ESSZETT in Unicode 5.0.


=>
The uppercase of ezsett in Unicode is "SS", following national standards and
practices. As of Unicode 5.1, an uppercase version of eszett became
available. Under the Unicode case folding, both map to "ss".


>
> Under IDNA2003, both DISALLOWED and UNASSIGNED characters
> are looked up. If abusive registrations are made using DISALLOWED
> or UNASSIGNED characters, these registered domain names may be
> be found on lookup by IDNA2003-compliant clients.


This is not correct, as Erik points out.


>
>
> Under IDNA2008, UNASSIGNED and DISALLOWED characters are not looked up.
> If new characters become defined under a new version of Unicode
> an old client will not look them up until it is updated. Abusive
> registrations
> using UNASSIGNED characters will not be looked up.
>
> Script mixing is not banned under IDNA2003. Under IDNA2008, BiDi
> bans mixing of European and Extended Arabic-Indic numbers with
> Arabic numbers.  That is AN and EN characters may not be present in
> the same label. Otherwise, mixing is permitted in IDNA2008.
>
> IMPLICATIONS OF ADOPTING IDNA2008 AS CURRENTLY SPECIFIED
>
>
> 1. IDNA2008 is case sensitive for labels with non-LDH characters in them
> but  is

... with at least one non-LDH character...

>
> case-insensitive for LDH characters
>
> for example" buecher "is all ASCII and could be matched with "Buecher" or
> "bUecher"
> under IDNA2008
>
> however "B<u-umlaut>cher" would not be allowed because Tables (see 4.2.2)
> would
> disallow Latin Capital letters. Some users accustomed to LDH-label behavior
> may be surprised that "B<u-umlaut>cher" and "b<u-umlaut>cher" do not match.
>
> On the other hand, the symmetric relationship between the IDNA2008-defined
> A-Label and U-Label has the benefit one can use exact match for either
> U-label form or A-label forms since they are directly and unambiguously
> transformable into each other.



> However, this symmetry will not exist for
>
> cases where the IDNA2003 A-Label and IDNA2008 A-label for the same
>
> U-Label differ. [Query: will this be a material problem only for actual
>
> registrations under IDNA2003 that differ in A-label form from IDNA2008?]
>
>
For registries, this is an advantage (equivalent to disallowing mapping),
but it is not so clear that it is a "benefit" for lookup.


>
>
> 2. IDNA2008 does not ban script mixing even within labels.
>
> Attempts to fashion rules along these lines have run into problems
> in which characters that may be confused for others are needed
> to express strings in particular languages. The International Phonetic
> Alphabet (IPA) characters are a case in point. Some are used for
> certain (e.g. African) languages but some of these characters
> can be confused for others in the Latin alphabet. Other examples
> exist in Arabic, Cyrillic, Greek among others.
>
> Even in the absence of intra-label script mixing, inter-script confusion
> such as the Russian word for "restaurant" looking like  "pectopah" in
> Latin characters is quite possible.
>
> Despite the apparent desirability of such a ban at protocol level, there
> are simply too many combinations of confusion within-scripts and between
> scripts to benefit significantly from a protocol-level ban. On the other
> hand,
> registry level constraints that may be more script-aware appear to be
> the most effective tool we have.


I think client-level warnings are the most effective constraint. After all,
if we could always trust the registries, we would need *no* constraints on
the client side in the protocol.


>
>
> 3. Esszet is permitted and its usage appears to be geographically and
> language
> specific. Under IDNA2003, this character is mapped into "ss". To deal with
> the
> potential conflict with previously mapped registrations in which Esszet is
> mapped
> to "ss" registries would need to appeal to Rationale 7.2 options, for
> example,
> to deal with this. Note that not all collisions may be a consequence of
> mapping, i.e.,
> many occurrences of "ss" in German text are not typographic variations of
> Esszett and very few occurrences in Latin script, without consideration of
> language,
> are variations of Esszett either.
>
> 4. Final Sigma is permitted and raises similar issues to Esszet with regard
> to
> collisions and the same remedies would apply.
>
> 5. ZWJ/ZWNJ
>
> In IDNA2003, these characters were mapped to "nothing". It has become
> apparent
> however that some Indic scripts need them. Persian registries currently
> reject registration of labels including ZWJ/ZWNJ although ZWNJ is used in
> writing Persian languages. Arabic language does not need ZWJ/ZWNJ.
> Mapping to "nothing" in INDA2003 has the side-
> effect of inhibiting domain name expression in some Indic scripts including
> Tamil and Devanagari. Permitting either or both as valid characters creates
> a compatibility problem similar to the Esszett one; i.e., one cannot tell
> whether a DNS label, when converted back to native character form, was
> intended to be written with ZWJ, ZWNJ or neither.
>
> Elaboration: Suppose that "ab" is a string in one of the scripts in which
> we now
> propose to permit ZWNJ.  All we have in the DNS is the A-label equivalent
> of "ab".
> We can't tell from looking at it whether the starting string, as
> seen/preferred by the
> registrant, was
>  ab    or
>  aZWJb
> since both map to the same A-label.
>
> Under IDNA2008, if the user enters "ab", she gets one A-label
> while, if she enters "aXWJb", she gets a different A-label.
> That is exactly the same as the Eszett problem -- you can't tell
> from the IDNA2003 A-label what the original intention was and
> use of the string under IDNA2008 gets you a different A-label
> than it does under IDNA2003.
>
> Joiner characters become invisible if inserted in strings written in
> scripts
> that do not use them.

=> in strings where they make no visual difference. This included scripts
that do not use them, and many positions in scripts that do use them.
>
> Unicode classifies these characters
> as "COMMON" so they also end up passing any plausible tests to prevent
> mixing of scripts in a label. Contextual rules are needed to restrict their
> use
> to strings in scripts where they have some effect.


where they could have some effect (they won't always, and even when they
commonly have an effect, it depends on the font).


> We end up relying on
> registries to adopt their use judiciously within those scripts. See also
> the
> Rationale document for further commentary.
>
> 6. Symbols and punctuation are NOT PVALID under IDNA2008 but are valid

Most symbols...

>
> under IDNA2003 leading to a variety of potential confusions with
> "slash-like"
> symbols or other symbols used in URIs for example. IDNA2008 rules reduce
> confusion potential by making all characters with these Unicode properties
> invalid for use with Domain labels.

either most, or add at the end "with certain exceptions"

>
>
> It is not clear that such symbols are critically needed for domain names.
>
> Another reason for banning these characters is that they complicate
> references, discussions and databases (such as WHOIS) because it is
> not clear how to describe them in common, informal usage.
> What is the correct way to refer to "-" ? Is it "hyphen", "minus sign",
> "hyphen-
> minus" or "short middle horizontal bar?" And is "." "period", "dot", "full
> stop",
> or something else? What about "#" - is it "pound", "hash", "number sign" or
> "tic-tac-toe"? "Heart" is another example: which one is it?


Thus is just not an issue; there are thousands of letters that have
ambiguous or multiple names. This paragraph just can't be fixed; it needs
removal.


>
>
> To be fair, one could refer to the Unicode long name for the character or
> even
> the "U+" form although this sounds pretty awkward in practical terms.
>
> 7. JAMO characters in Korean have been made Protocol Invalid (DISALLOWED)
> for reasons similar to (6) above. They introduce a combinatorial explosion
> of different
> string representations built from JAMO primitive characters. They are valid
> under IDNA2003.


This is debatable. The only reasonable rationale we can include is that they
are only used in historic Korean.

>
>
> 8. Under INDA 2008, when a new version of Unicode is released the following
> steps can be taken:
>
> a. review of changes that might require new rules in the IDNA2008
> framework.
> Such a conclusion would assuredly require formation of a  WG to facilitate
> \new RFC
> production. This is thought to be extremely unlikely to happen.
>
> b. A review of changes might only require exception rules to preserve
> compatibility. It is possible that the required changes might be delegated
> to an IANA action possibly in consultation with an expert committee
> to generate new tables.


The current drafts require new RFCs in order to change the exception tables,
I believe. It would be better to change that to have the exception table
governed by the same process as the context tables (under stability
provisions).


>
> c. Generate new tables for IANA registry (suitable for downloading as
> needed
>
> During the transition some will clients have the older tables and
> some registries the newer ones. Lookups of Domain Names containing
> new PVALID characters by older clients will fail under IDN2008 because
> the client will reject UNASSIGNED characters until the clients are updated
> with the new PVALID characters.


That is not the bad part of the transition. The bad part is that old
characters may transform from DISALLOWED to PVALID only during the
transition, then corrected, or transform from PVALID to DISALLOWED only
during the transition, then corrected. And the correction period may be
long, depending on when software is updated. That is, if a program ships
every two years, and is updated during the correction, it will be wrong for
2 years.


> ...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.alvestrand.no/pipermail/idna-update/attachments/20090322/e6a35760/attachment-0001.htm