<br clear="all">Mark<br>
<br><br><div class="gmail_quote">On Tue, Jun 30, 2009 at 03:29, "Martin J. Dürst" <span dir="ltr"><<a href="mailto:duerst@it.aoyama.ac.jp" target="_blank">duerst@it.aoyama.ac.jp</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><br>
<br>
On 2009/06/30 2:55, John C Klensin wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Mark,<br>
<br>
Several comments inline...<br>
<br>
--On Sunday, June 28, 2009 21:28 -0700 Mark Davis ⌛<br>
<<a href="mailto:mark@macchiato.com" target="_blank">mark@macchiato.com</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Returning to the discussion, now that some of my other<br>
standards work is under control (RFC4646bis was approved,<br>
whew!)<br>
...<br>
</blockquote>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Now, my position is still that the simplest and most<br>
compatible option open to us is to simply map with NFKC +<br>
Casefold.<br>
</blockquote>
<br>
I continue to believe that CaseFold is a showstopper. When its<br>
results are not identical to those produced by LowerCase, it<br>
produces results that are astonishing to some users and leads us<br>
into the "is that a separate character or not" trap that we've<br>
seen manifested at least twice. I note that TUS recommends<br>
against its use for mapping (as distinct from comparison) and<br>
appears to do so for just the reason that it involves too much<br>
information loss.<br>
</blockquote>
<br></div>
I have earlier said that I think Mark's proposal goes in the right direction, but I agree with John that LowerCase is better than CaseFold. If anything, the burden of proof should be on the CaseFold side (show, for each case of mapping that's in CaseFold but not LowerCase, why it's needed) rather than on the LowerCase side.</blockquote>
<div><br>The key issue for me is that we don't map <i>differently<b> -- unless it is vital --</b></i> from how IDNA2003 mapped. Different mappings are <i><b>very</b></i> painful for implementations, and will be very painful for years to come for users.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<br>
Mark wrote, in a later mail:<div><br>
<br>
You make it sounds like final sigma, ZWJ/NJ, eszett and the other cases<br>
under discussion were oversights in the process of developing the current<br>
IDNA. That wasn't the case; these were deliberate choices made at the time.<br>
A case mapping is also a 'loss of information', but one that people clearly<br>
want.<br>
<br>
<br></div>
Eszett wasn't exactly an oversight, I knew at the time that it was problematic and told others. However, I didn't have the zeal to defend it because as a Swiss, I didn't and don't feel as attached to it as Germans and Austrians do.<br>
<br>
My understanding of why the eszett got mapped in IDNA 2003 was that the IETF wanted a one-stop shopping table, and Unicode had such a table, and any discussions about individual characters were out of fashion because it was felt that if we started discussing individual characters, we would never finish.</blockquote>
<div><br>The main reason, as I recall, for favoring the mapping to "ss" is because then the uppercases match correctly. But we'd have to go back and dredge through all the email to see get any feeling for what the decisive arguments were. There are certainly a number of exceptional cases in IDNA2003 that are called out in different categories...<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><br>
<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
...<br>
Proposal: A. Tables document<br>
<br>
Add a new type of character: REMAP. A character is REMAP if it<br>
meets *all of * the following criteria:<br>
<br>
1. The character is not PVALID or CONTEXTO<br>
2. If remapped by the Unicode property NFKC_Casefold*, then<br>
the resulting character(s) are all PVALID or CONTEXTO<br>
3. The character is a LetterDigit or Pd<br>
4. The character has one of the following<br>
Decomposition_Type values: initial, medial, final,<br>
isolated, wide, narrow, or compat<br>
</blockquote>
<br>
I am very concerned that collapsing initial, medial, and final<br>
together may get us into problems with other language<br>
communities similar to those we have gotten into with Final<br>
Sigma, especially when those communities denote word boundaries<br>
by the appearance of final or initial forms and hence would use<br>
those forms in a style similar to the way "BigCompany" or<br>
"big-company" might be used in ASCII.<br>
</blockquote>
<br></div>
The only character currently not containing the word "ARABIC" in its name for <initial>, <medial>, <final>, or <isolated> is U+FDFC, RIAL SIGN, which is just as well Arabic even if it doesn't say so in its name.<br>
<br>
I strongly doubt that the UTC would encode other backwards compatibility contextual forms in these four categories, and it might be possible to make sure that doesn't happen with a stability guarantee if that's really necessary.<br>
<br>
What I already asked Mark for, and what I'm still looking for, is some data on how (in)frequent these actually are.</blockquote><div><br>These are infrequent, as far as my initial sampling goes. I can't really release all data (sorry for not answering before), and for the less-frequent cases I really have to get a much larger sample in order to have meaninful statistics. Here's an example:<br>
<br>U+FEA9 ( ﺩ ) ARABIC LETTER DAL ISOLATED FORM<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<br>
<br>
As for <wide>, that includes only U+3000 (full width space, irrelevant here) and U+FFxx characters that contain FULLWIDTH in their name.<br>
<br>
As for <narrow>, that includes HANGUL, KATAKANA, and 11 characters in the U+FFxx area, all of which contain the word HALFWIDTH. The one to watch out for is U+FF61, HALFWIDTH IDEOGRAPHIC FULL STOP. Its fullwidth sibling (U+3002) is part of IDNA 2003.<br>
<br>
For these two (wide/narrow), I know from local experience here in Japan that they are probably necessary. Still, it would be good to get some numbers from Mark.</blockquote><div><br>The highest frequency width-variant characters are:<br>
<br>U+FF70 ( ｰ ) HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK<br>U+FF83 ( ﾃ ) HALFWIDTH KATAKANA LETTER TE <br>...<br>U+FF0D ( － ) FULLWIDTH HYPHEN-MINUS<br>U+FF43 ( ｃ ) FULLWIDTH LATIN SMALL LETTER C<br>
...<br><br>The fullwidth hyphen is the highest frequency character that would be subject to remapping under the proposed scheme; so higher frequency than any case-mapped character outside of ASCII. The top 4 characters are:<br>
<br>U+FF0D ( － ) FULLWIDTH HYPHEN-MINUS<br>U+00AD ( ) SOFT HYPHEN<br>U+00C3 ( Ã ) LATIN CAPITAL LETTER A WITH TILDE<br>U+00DF ( ß ) LATIN SMALL LETTER SHARP S<br><br>Note that soft-hyphen is mapped away by IDNA2003. It is invisible, and only serves to indicate hyphenation points, so it is easy for someone to cut and paste a word into an href, for example, without knowing that it is there.<br>
<br>The eszett is not included in REMAP because it is PVALID.<br><br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<br>
<br>
As for <compat>, that's the "everything else" bucket. That's a total of 720 characters in Unicode 5.2 (as of UnicodeData-5.2.0d9.txt). Not all of them qualify by Mark's rules (in particular things such as parenthesized numbers don't because parentheses aren't allowed), but there are still way to many in my opinion that qualify. It would be good to know from Mark how many of these he really thinks need to be mapped, and why. If that's let's say 90% or 95% of the characters that would qualify by Mark's rules, it might be okay to just leave the rest as is, provided we can see no harm. Otherwise, I think a more detailed analysis may be necessary.<br>
<br>
To be more explicit, I think *at least* the following are included by the rules that Mark proposes but shouldn't be used for mapping:<br>
<br>
- ROMAN NUMERALs (32)<br>
- CJK/KANGXI RADICALs (216)<br>
- IDEOGRAPHIC TELEGRAPH SYMBOLs (68)</blockquote><div><br>I put the exact cases generated on my page; the cases you are worried about are not there. There are 59 characters, and they don't include the above. The characters are:<br>
<a href="http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%C2%B5" target="_blank">http://unicode.org/cldr/utility/list-unicodeset.jsp?a=[µ</a>IJijĿŀʼnſDŽ-njDZ-dzϐ-ϒϕϖϰ-ϲϵϹևٵ-ٸำຳໜໝ\u0F77\u0F79ẚℇℵ-ℸff-stﬓ-ﬗﭏ]<br>
You can look at details on the other categories in the breakdown with the same tool.<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
Excluding characters with the words HANGUL, PARENTHESIZED, COMMA, and FULL STOP (all of which are excluded by Mark's rules) reduces the overall total from 720 to 456. In these, there are at least three categories:<br>
- Some more that are already excluded my Mark's rules but that my simple greps didn't catch.<br>
- Those that I think definitely shouldn't be included (see above, 316 in total)<br>
- The rest, possibly okay to include, which is at most 140.<div><br>
<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
As I've said several times before, even if we disallow the<br>
NFKC-affected forms those characters, if a need arises, we can<br>
(painfully) redefine them as PVALID and allow them. But, if we<br>
map them to something else, we lose all information about what<br>
was intended/desired and end up in precisely the mess we have<br>
with e.g., Final Sigma and ZWJ/ZWNJ in which "the right thing<br>
to do" poses enough compatibility problems to cause opposition<br>
to making changes.<br>
</blockquote>
<br></div>
We definitely have to look at this carefully. I'm not overly concerned in general, but we shouldn't just gloss over it.</blockquote><div><br>I agree. Feedback on the data would be appreciated.<br> <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
5. The character does not have the Script value: Hangul<br>
<br>
The REMAP characters are removed from DISALLOWED, so that the<br>
TABLES values form a partition (all the values are disjoint).<br>
</blockquote>
<br>
This strikes me as dangerous -- see below.<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
B. Protocols documentChange sections 4.2.1 and 5.3 so as to<br>
require:<br>
<br>
1. Mapping all REMAP characters according to the Unicode<br>
property NFKC_Casefold,<br>
2. Then normalizing the result according to NFC.<br>
</blockquote></blockquote>
<br></div>
We have to make sure this transform is idempotent on all strings we are concerned about, or introduce additional steps if necessary.</blockquote><div><br>It is, but if we are paranoid it is easy to ensure idempotence in the spec with a "repeat until there is no change clause". <br>
<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<br>
Regards, Martin.<div><div></div><div><br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Making this change to 4.2.1 eliminates the requirement that the<br>
registrant understand _exactly_ what is being registered, i.e.,<br>
that the communication path between the registrant and registry<br>
occur only using U-labels and/or A-labels. My understanding was<br>
that we had reached one of the more clear consensus we had in<br>
these discussions that the "no mapping on registration"<br>
restriction was appropriate. Are you proposing to reopen that<br>
question?<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
The rest of the tests for U-Label remain unchanged.<br>
</blockquote>
<br>
I believe that doing this by the type of change to Tables that<br>
you recommend either requires a change to the way that the<br>
definition of U-label is stated or requires us to abandon the<br>
very clear concept of a U-label that is completely symmetric,<br>
with no information loss in either direction, with an A-label.<br>
<br>
There is also a subtle interaction with Section 5.5: if the<br>
mapping is performed by the time Section 5.3 concludes (or,<br>
under special circumstances, not applied at all), then Section<br>
5.5 must also prohibit REMAP.<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
C. Defs document<br>
<br>
1. Define REMAP<br>
2. Define an M-Label to be one which if remapped according<br>
to B1+B2, results in a U-Label.<br>
</blockquote>
<br>
The idea of an M-Label still makes me uncomfortable. Again, we<br>
have had that discussion before.<br>
<br>
regards,<br>
john<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Idna-update mailing list<br>
<a href="mailto:Idna-update@alvestrand.no" target="_blank">Idna-update@alvestrand.no</a><br>
<a href="http://www.alvestrand.no/mailman/listinfo/idna-update" target="_blank">http://www.alvestrand.no/mailman/listinfo/idna-update</a><br>
</blockquote>
<br></div></div><div><div></div><div>
-- <br>
#-# Martin J. Dürst, Professor, Aoyama Gakuin University<br>
#-# <a href="http://www.sw.it.aoyama.ac.jp" target="_blank">http://www.sw.it.aoyama.ac.jp</a> mailto:<a href="mailto:duerst@it.aoyama.ac.jp" target="_blank">duerst@it.aoyama.ac.jp</a><br>
</div></div></blockquote></div><br>