Returning to the discussion, now that some of my other standards work is under control (RFC4646bis was approved, whew!)<br><br><hr size="2" width="100%"><h2><a name="TOC-Details-on-REMAP"></a>
</h2>I've had a chance to do some data mining, and it is now clear which are
the most prominent characters that are remapped under the current
scheme (the relative frequencies vary, as one might expect, depending
on the language): they are case variants, width variants, and
presentation variants.<br><br>A copy of this email is at <a href="http://www.macchiato.com/unicode/idna/remap" rel="nofollow">http://www.macchiato.com/unicode/idna/remap</a><br>
<br>
Now, my position is still that the
simplest and most compatible option open to us is to simply map with
NFKC + Casefold. However, in the interest of getting this process
moving, I offer the following as a possible compromise approach. It
limits the remapped characters to those that are the most useful in
practice. I'll first give the proposal, then list some of the details
afterwards.<br><h2><a name="TOC-Proposal:"></a>
Proposal:</h2>
<h3><a name="TOC-A.-Tables-document"></a>
A. Tables document</h3><p>Add a new type of character: REMAP. A character is REMAP if it meets <b>all of</b> the following criteria:<br>
</p>
<div><ol><li>The character is not PVALID or CONTEXTO</li><li>If remapped by the Unicode property NFKC_Casefold*, then the resulting character(s) are all PVALID or CONTEXTO</li><li>The character is a LetterDigit or Pd<br>
</li><li>The character has one of the following Decomposition_Type values: initial, medial, final, isolated, wide, narrow, or compat</li><li>The character does not have the Script value: Hangul</li></ol>
The REMAP characters are removed from DISALLOWED, so that the TABLES values form a partition (all the values are disjoint).<br>
<br><h3><a name="TOC-B.-Protocols-document"></a>
B. Protocols document</h3>Change sections 4.2.1 and 5.3 so as to require:<ol><li>Mapping all REMAP characters according to the Unicode property NFKC_Casefold,</li><li>Then normalizing the result according to NFC.</li></ol>
The rest of the tests for U-Label remain unchanged.<br>
<br><h3><a name="TOC-C.-Defs-document"></a>
C. Defs document</h3>
<ol><li>Define REMAP</li><li>Define an M-Label to be one which if remapped according to B1+B2, results in a U-Label.<br>
</li></ol>
<br><hr size="2" width="100%"><h2><a name="TOC-Details-on-REMAP"></a>
Details on REMAP</h2>
<ol><li>The character is not PVALID or CONTEXTO</li><ul style="background-color: rgb(153, 255, 153);"><li>This guarantees that the class of REMAP is disjoint with the others<br></li></ul><li>If remapped by the Unicode property NFKC_Casefold*, then the resulting character(s) are all PVALID or CONTEXTO</li>
<ul style="background-color: rgb(153, 255, 153);"><li>This condition is not strictly necessary. Because of the way in which REMAP is used in the protocol above,
if a character results that is not PVALID, then it would fail the later
tests. So as far as I'm concerned, this could be dropped. However,
restricting the characters in this way will probably make a character listing clearer to people.</li><li>The derived property <a name="NFKC_Casefold">NFKC_Casefold is being added to Unicode 5.2, and is already present in the 5.2 beta. </a>It
provides a convenient way to fold characters for identifiers (and not
just for IDNA). It is defined in
<a href="http://unicode.org/reports/tr44/tr44-3.html" rel="nofollow">http://unicode.org/reports/tr44/tr44-3.html</a>, and the characters
affected are listed in <a href="http://unicode.org/Public/5.2.0/ucd/" rel="nofollow">http://unicode.org/Public/5.2.0/ucd/</a>
under
DerivedNormalizationProps. If we didn't want to wait for U5.2, we can
define it on our own, but it would be convenient to use it, as long as
we release in October or later.<br></li></ul><li>The character is a LetterDigit or Pd<br></li><ul style="background-color: rgb(153, 255, 153);"><li><span style="background-color: rgb(153, 255, 153);">This limits the input characters by eliminating symbols, punctuation, etc.</span></li>
<li><span style="background-color: rgb(153, 255, 153);">The Pd is in only to pick up the fullwidth hyphen.</span></li></ul><li>The character has one of the following Decomposition_Type values: initial, medial, final, isolated, wide, narrow, or compat</li>
<ul style="background-color: rgb(153, 255, 153);"><li style="background-color: rgb(153, 255, 153);">The initial/medial/final/isolated forms are all Arabic presentation forms, such as:<code><a href="http://unicode.org/cldr/utility/character.jsp?a=FE91" rel="nofollow"><br>
U+FE91</a></code> ( ﺑ ) ARABIC LETTER BEH INITIAL FORM</li><li style="background-color: rgb(153, 255, 153);">The narrow/wide are all width variants, such as:<br>
<code><a href="http://unicode.org/cldr/utility/character.jsp?a=FF71" rel="nofollow">U+FF71</a></code> ( ｱ ) HALFWIDTH KATAKANA LETTER A</li><li style="background-color: rgb(153, 255, 153);">The compat forms include various digraphs or forms such as:<br>
<code><a href="http://unicode.org/cldr/utility/character.jsp?a=013F" rel="nofollow">U+013F</a></code> ( Ŀ ) LATIN CAPITAL LETTER L WITH MIDDLE DOT,<br>
<code><a href="http://unicode.org/cldr/utility/character.jsp?a=01C6" rel="nofollow">U+01C6</a></code> ( dž ) LATIN SMALL LETTER DZ WITH CARON<br>
Most compat characters are, however, eliminated by other conditions, such as:<br>
<code><a href="http://unicode.org/cldr/utility/character.jsp?a=2474" rel="nofollow">U+2474</a></code> ( ⑴ ) PARENTHESIZED DIGIT ONE, which is eliminated by both condition #2 and #3<br>
</li><li><code style="background-color: rgb(153, 255, 153);"></code><span style="background-color: rgb(153, 255, 153);">The excluded decomposition types are: font, super, sub; vertical; circle, fraction, nobreak, small, square</span><br>
</li></ul><li>The character does not have the Script value: Hangul</li><ul style="background-color: rgb(153, 255, 153);"><li>This is consistent with the exclusion in Tables of OldHangulJamo. Sample exclusions are:<br>
<code><a href="http://unicode.org/cldr/utility/character.jsp?a=3131" rel="nofollow">U+3131</a></code> ( ㄱ ) HANGUL LETTER KIYEOK<br>
<code><a href="http://unicode.org/cldr/utility/character.jsp?a=FFA1" rel="nofollow">U+FFA1</a></code> ( ﾡ ) HALFWIDTH HANGUL LETTER KIYEOK</li></ul></ol>
</div><br><br><div class="gmail_quote">On Sun, Jun 7, 2009 at 13:49, Paul Hoffman <span dir="ltr"><<a href="mailto:phoffman@imc.org">phoffman@imc.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">At 11:57 AM -0400 6/7/09, John C Klensin wrote:<br>
>--On Saturday, June 06, 2009 16:38 -0700 Paul Hoffman<br>
><<a href="mailto:phoffman@imc.org">phoffman@imc.org</a>> wrote:<br>
><br>
>>...<br>
>>> I continue to believe that use of NKFC without exclusion of<br>
>>> character groups for which there are no justifications is<br>
>><br>
>> Pete's proposed mapping happens before the<br>
>> is-it-valid-IDNA2008 check. Why should we use a modified NFKC<br>
>> instead of plain-vanilla-NFKC and let the second step<br>
>> (is-it-valid-IDNA2008 check) happen as-is?<br>
><br>
>My concern is not those NFKC mappings that will result in<br>
>invalid (DISALLOWED) characters. It is<br>
><br>
>(1) NFKC mappings of characters that, if used in domain names,<br>
>are probably used to cause mischief and for which there is no<br>
>substantive justification. The "Mathematical" characters are<br>
>examples of this.<br>
<br>
</div>I'm still confused. If someone enters a mathematical character that is mapped to a allowed character, the result is a valid domain name that could have been entered as allowed characters. This is identical to what we have today in IDNA2003, no worse.<br>
<div class="im"><br>
> Martin's original list identified others.<br>
>Note that, except in specialized systems, these characters are<br>
>very difficult to type and ones for which fonts are unlikely to<br>
>be present.<br>
<br>
</div>Yes, exactly. I'm still missing your point of concern.<br>
<div class="im"><br>
>(2) NFKC mappings of characters that result in characters in<br>
>CONTEXTO or CONTEXTJ. Unless I missed something in my search,<br>
>this is a null set at present. But I can find no stability rule<br>
>that would prevent adding such a character and the same<br>
>presentation and ambiguity issues that apply to the listed<br>
>CONTEXTx characters would apply to their compatibility<br>
>equivalents.<br>
<br>
</div>And I don't see a problem with that. Someone enters an name-which-needs-mapping, it is mapped, and out pops some characters that are valid. How is this of more concern than a valid, no-mapping IDNA2008 name?<br>
<div class="im"><br>
> >> (i) A violation of the "inclusion" model of IDNA2008<br>
>><br>
>> Completely agree. However, this whole document is a violation<br>
>> of the "no-mapping" model of IDNA2008, so that seems like an<br>
>> odd objection.<br>
><br>
>We are likely to have to agree to disagree about this, but I<br>
>believe that "inclusion" and "no mapping" are separate<br>
>principles. The acceptance of mapping in some contexts does not<br>
>seem to me to justify, in any way, abandoning "inclusion". From<br>
>that point of view, the argument to abandon inclusion in the<br>
>mapping context has to be made separately... and that argument<br>
>has not, as far as I can remember, been made yet.<br>
<br>
</div>We do agree to disagree here. Up to this point, we have held the whole set of rules constant, and some of them intertwined. I do *not* want us to abandon any of the rules in the end, just to allow sensible mapping before doing the final rule check.<br>
<div class="im"><br>
> >> (ii) A violation of the closely-related protocol design<br>
>>> principle that one should include only those things for which<br>
>>> one has both use and understanding because it is easier to add<br>
>>> later than it is to remove.<br>
>><br>
>> Implementers of IDNA2008 will understand NFKC as well as<br>
>> implementers of IDNA2003.<br>
><br>
>Which is to say, IMO, not at all.<br>
<br>
</div>And here we agree. Thus, I see no harm in extending the protocol. We have not seen any significant damage from the lack of understanding in IDNA2003 names, just as the lack of understanding of some crypto properties (for example) in developers hasn't had any significant negative effect on, say IPsec and TLS. And I really do see a parallel between the two types of protocols.<br>
<div class="im"><br>
> >> (iii) An increased risk, however slight, that we will, in the<br>
>>> future, get strong demands from some particular community to<br>
>>> treat a character classified by Unicode as "compatibility" as<br>
>>> a real and distinct character. If such a character is<br>
>>> disallowed by virtue of not being mapped, we will have the<br>
>>> difficult problem of changing a disallowed character to a<br>
>>> PVALID one. But, if it is mapped to something else, we will<br>
> >> have to revisit the very complex discussions that we have had<br>
>>> over Eszett and Final Sigma. We should not incur that risk<br>
>>> unless there is a reason to do so.<br>
>><br>
>> There is: increased (but, of course, not full) backwards<br>
>> compatibility with the large installed base of IDNA2003.<br>
><br>
>We've seen no evidence that any of these other categories of<br>
>compatibility characters are used --other than in possible<br>
>demonstrations or out of malice-- in IDNA2003-compatible<br>
>applications, much less enough to constitute a large installed<br>
>base.<br>
<br>
</div>Really? I thought that multiple presentations by Asians showed that some of the compatibility characters got entered automatically by the UIs of some browsers and so on. I could be mistaken, of course, but that is certainly what I interpret slide 3 of <<a href="http://www.ietf.org/proceedings/09mar/slides/idnabis-4.pdf" target="_blank">http://www.ietf.org/proceedings/09mar/slides/idnabis-4.pdf</a>>.<br>
<br>
>YMMD.<br>
<br>
As we age, all of our milages are degrading, yes. :-)<br>
<div><div></div><div class="h5">_______________________________________________<br>
Idna-update mailing list<br>
<a href="mailto:Idna-update@alvestrand.no">Idna-update@alvestrand.no</a><br>
<a href="http://www.alvestrand.no/mailman/listinfo/idna-update" target="_blank">http://www.alvestrand.no/mailman/listinfo/idna-update</a><br>
</div></div></blockquote></div><br>