Comments on IDNAbis issues-05

Mark Davis mark.davis at
Fri Dec 14 04:48:37 CET 2007
Overview.Many nice improvements to the text.

*This list does not list what was already commented on for **
* and **draft-faltstrom-idnabis-tables-03.txt*<>

Issues-1. IDNAbis has a major backwards compatibility issue with IDNA2003:
thousands of characters are excluded that used to be valid. What reason
might people have to believe that despite the terms NEVER and ALWAYS that
some future version, IDNAbis-bis, might not also do the same?

Issues-2. IDNA provided for backwards compatibility, by disallowing
Unassigned characters in registration, but allowing them in lookup. That let
old clients work despite new software. While once we update to U5.1 that is
not as much of a problem, it should be made clear why this change is made.

Issues-3. In general, whenever a statement is made about some class of
characters causing a problem, at least one clear example should be provided,
as in draft-alvestrand-idna-bidi-01.txt<>

Issues-4. I would strongly suggest separating all of the "why did we do
this" and "how is it different from IDNA2003" into a separate document. It
will be of only historical interest after this becomes final, and will then
only clutter the document.


   IDNA uses the Unicode character repertoire, which avoids the
   significant delays that would be inherent in waiting for a different
   and specific character set be defined for IDN purposes, presumably by
   some other standards developing organization.

Seems odd. There are no other contenders in the wings. Would be better, if
this has to be said, to just cite other IETF documents describing the
reasons for using Unicode.


   To improve clarity, this document introduces three new terms.  A
   string is "IDNA-valid" if it meets all of the requirements of this
   specification for an IDNA label.  It may be either an "A-label" or a
   "U-label", and it is expected that specific reference will be made to
   the form appropriate to any context in which the distinction is
   A "U-label" is an IDNA-valid string of
   Unicode-coded characters that is a valid output of performing
   ToUnicode on an A-label, again regardless of how the label is
   actually produced.

These definitions appear circular, so they need to be teased out a bit.


   Depending on the system involved, the major difficulty may not lie in
   the mapping but in accurately identifying the incoming character set
   and then applying the correct conversion routine.  It may be
   especially difficult when the character coding system in local use is
   based on conceptually different assumptions than those used by
   Unicode about, e.g., how different presentation or combining forms
   are handled.  Those differences may not easily yield unambiguous
   conversions or interpretations even if each coding system is
   internally consistent and adequate to represent the local language
   and script.

I suggest the following rewrite:

The main difficulty typically is that of  accurately identifying the
incoming character set so as to apply the correct conversion routine.
Theoretically, conversion could be difficult if the non-Unicode character
encoding system were based on conceptually different assumptions than those
used by Unicode about, e.g., how different presentation or combining forms
are handled. Some examples are the so-called "font-encodings" used on some
Indian websites. However, in modern software, such character sets are rarely
used except for specialized display.


   That, in turn, indicates that the script community
   relevant to that character, reflecting appropriate authorities for
   all of the known languages that use that script, has agreed that the
   script and its components are sufficiently well understood.  This
   subsection discusses characters, rather than scripts, because it is
   explicitly understood that a script community may decide to include
   some characters of the script and not others.

   Because of this condition, which requires evaluation by individual
   script communities of the characters suitable for use in IDNs (not
   just, e.g., the general stability of the scripts in which those
   characters are embedded) it is not feasible to define the boundary
   point between this category and the next one by general properties of
   the characters, such as the Unicode property lists.

There is no justification given for this process. Moreover, it will be
doomed to failure. Merely the identification of "script communities" is an
impossible task. Who speaks for the Arabic script world? Saudi Arabia
(Arabic)? Iran (Persian,...)? Pakistan (Urdu,...)?, China (Uighur,...)?


      it is removed from Unicode.

(multiple instances)

This is not necessary; characters aren't removed from Unicode. If you really
have to have it, then add "(however, the Unicode stability policies
expressly forbid this)"


   Applications are expected to not treat "ALWAYS" and "MAYBE"
   differently with regard to name resolution ("lookup").  They may
   choose to provide warnings to users when labels or fully-qualified
   names containing characters in the "MAYBE" categories are to be

In practice, expecting applications to treat these differently is wishful
thinking; especially if it seems Eurocentric to users (see other notes on
MAYBE). In practice, registries always have the ability to filter characters
out. See above on removing Maybe.



I know what the point is supposed to be (and don't disagree), but this
section was very hard to make out.


   Characters that are placed in the "NEVER" category are never removed
   from it or reclassified.  If a character is classified as "NEVER" in
   error and the error is sufficiently problematic, the only recourse is
   to introduce a new code point into Unicode and classify it as "MAYBE"
   or "ALWAYS" as appropriate.

The odds of this happening are extremely low. Anything in Never has to be
extremely certain.


   Instead, we need to have a variety of approaches that, together,
   constitute multiple lines of defense.

Defense against what? Without examples, it is hard to say what the problems


   Applications MAY
   allow the display and user input of A-labels, but are not encouraged
   to do so except as an interface for special purposes, possibly for
   debugging, or to cope with display limitations.  A-labels are opaque
   and ugly, and, where possible, should thus only be exposed to users
   who absolutely need them.  Because IDN labels can be rendered either
   as the A-labels or U-labels, the application may reasonably have an
   option for the user to select the preferred method of display; if it
   does, rendering the U-label should normally be the default.


 It is, however, now common practice to display a suspect U-Label (such as a
mixture of Latin and Cyrillic) as an A-Label.


   6.3.  The Ligature and Digraph Problem

   There are a number of languages written with alphabetic scripts in
   which single phonemes are written using two characters, termed a

   "digraph", for example, the "ph" in "pharmacy" and "telephone".

The text has been improved considerably from earlier versions, but the whole
issue is just a special case of the fact that words are spelled different
ways in different languages or language variants. And it has really nothing
to do with ligatures and diagraphs. The same issue is exhibited between and as between a Norwegian URL with ae and a Swedish
one with a-umlaut.

So if you retain this section, it should be recast as something like

6.3 Linguistic Expectations

 Users often have certain expectations based on their language. A Norwegian
user might expect a label with the ae-ligature to be treated as the same
label using the Swedish spelling with a-umlaut. A user in German might
expect a label with a u-umlaut and the same label with "ae" to resolve the
same. For that matter, an English user might expect "" and "" to resolve the same. [more in that vein].


 there is no evidence that
they are important enough to Internet operations or
internationalization to justify large numbers of special cases
and character-specific handling (additional discussion and

I suggest the following wording instead:

 there is no evidence that
 they are important enough to Internet operations or
 internationalization to justify inclusion (additional discussion and

It doesn't actually involve "large numbers of special cases", there are a
rather small percentage of demonstrable problems in the symbol/punctuation
area. What we could say is that there is general consensus that removing all
but letters, digits, numbers, and marks (with some exceptions) causes little
damage in terms of backwards compatibility, and does remove some problematic
characters like fraction slash.


   For example, an essential
   element of the ASCII case-mapping functions is that
   uppercase(character) must be equal to

Remove or rephrase. It is a characteristic, but not an essential one. In
fact, case mappings of strings are lossy; once you lowercase "McGowan", you
can't recover the original.


   o  Unicode names for letters are fairly intuitive, recognizable to
      uses of the relevant script, and unambiguous.  Symbol names are
      more problematic because there may be no general agreement on
      whether a particular glyph matches a symbol, there are no uniform
      conventions for naming, variations such as outline, solid, and

Actually, the formal Unicode names are often far from intuitive to users of
the relevant script. That's because the constraints of using ASCII for the
name, to line up with ISO standards for character encodings.

This section is not really needed. The use of I<heart> is not really
problematic; the main justification for removing it is that we don't think
it is needed (and has not been used much since IDNA was introduced). Better
to just stick with that.


   11.  IANA Considerations

   11.1.  IDNA Permitted Character Registry

   The distinction between "MAYBE" code points and those classified into
   "ALWAYS" and "NEVER" (see Section 5) requires a registry of
   characters and scripts and their categories.  IANA is requested to

Expecting an IANA registry to maintain this is setting it up for failure. If
this were to be done, precise and lengthy guidance as to the criteria for
removing characters (moving to NEVER) would have to be supplied, because of
the irrevocable nature of this step. The odds of a registry being able to
perform this correctly are very small.

The best alternative would be to simply have all the non-historic scripts
have the same status in
by moving the non-historic scripts to the same status as Latin, Greek, and

The second best would be to have the Unicode consortium make the
determinations (and take the heat for objections).


   Some specific suggestion
   about identification and handling of confusable characters appear in
   a Unicode Consortium publication [???]

Use: [UTR36]
      UTR #36: *Unicode Security Considerations*
-------------- next part --------------
An HTML attachment was scrubbed...

More information about the Idna-update mailing list