I'll try to be brief.<br><br>1. Just some points of information (you may be aware of this, but for others' sakes):<br><br>1a. Letter Modifiers are not the same as characters having "MODIFIER LETTER" in their name. They overlap by 143 code points, but are different by 108 code points. See:<br>
<br> <a href="http://unicode.org/cldr/utility/unicodeset.jsp?a=[:lm:]&b=[:name=/MODIFIER%20LETTER/">http://unicode.org/cldr/utility/unicodeset.jsp?a=[:lm:]&b=[:name=/MODIFIER%20LETTER/</a>:]<br><br>(clicking on the Only in A, Only in B, and In both A and B takes you to the detail links.)<br>
<br>1b. Characters that are really not used in orthographies are already separated by property into the Sk group, which are already not being included in IDNA2008.<br><br><a href="http://unicode.org/cldr/utility/list-unicodeset.jsp?a=[:sk">http://unicode.org/cldr/utility/list-unicodeset.jsp?a=[:sk</a>:]<br>
<br>2. One could go through all of the Lm characters and do a detailed analysis of whether each would be needed or not, but that would take sizable time and effort, and be (of course) contentious. Unless we bring in experts it is quite difficult to tell what the usage of particular characters is. Moreover, Lm characters are not that much different in kind than normal letters, of which there are some 100,000 to go through. It is quite tricky to determine modern usage: cf <a href="http://www.macchiato.com/unicode/historic-unicode-characters">http://www.macchiato.com/unicode/historic-unicode-characters</a><br>
<br>3. We've always known that IDNA2008 simply cannot solve the visual confusability problem. Checking the Lm characters would be only nibbling at the edges; it really doesn't touch the 99.99% cases. And trying to do this in the protocol is a very heavy hammer - browsers and other clients need to do far more sophisticated contextual analysis than what the protocol can (or should) supply; they also have far more information available, such as the user's display language. Moreover, they can take action that the protocol can't, like warning about suspicious-looking URLs but allowing users to continue who what to.<br>
<br>4. The only reason I proposed Tatweel is that there is a fundamental difference: we encoded Tatweel specifically for its display effect, no other reason.<br><br clear="all">Mark<br>
<br><br><div class="gmail_quote">On Fri, Mar 20, 2009 at 10:26, John C Klensin <span dir="ltr"><<a href="mailto:klensin@jck.com">klensin@jck.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
--On Thursday, March 19, 2009 13:44 -0700 Mark Davis<br>
<<a href="mailto:mark@macchiato.com">mark@macchiato.com</a>> wrote:<br>
<br>
> I propose that we make U+0640 ( ـ ) ARABIC TATWEEL (aka<br>
> kashida) be DISALLOWED, adding it to<br>
> <a href="http://tools.ietf.org/html/draft-ietf-idnabis-tables-05#sectio" target="_blank">http://tools.ietf.org/html/draft-ietf-idnabis-tables-05#sectio</a><br>
> n-2.6. Currently it is PVALID, but it does not carry semantics<br>
> by any Arabic-Script orthography, and its only value is for<br>
> spoofing.<br>
><br>
> For example: جوجل can be written with extra kashidas as<br>
> جـوجل or as جوجـل by inserting a kashida after the<br>
> first or third character. This is very hard for users to<br>
> detect. We added it to Unicode for use in manual<br>
> justification, but has no place in IDNA.<br>
><br>
> (<a href="http://en.wikipedia.org/wiki/Kashida" target="_blank">http://en.wikipedia.org/wiki/Kashida</a>,<br>
> <a href="http://unicode.org/cldr/utility/character.jsp?a=0640" target="_blank">http://unicode.org/cldr/utility/character.jsp?a=0640</a>)<br>
<br>
Mark,<br>
<br>
First of all, I'm sympathetic to this recommendation and note<br>
that the ASIWG recommended to us last fall that the character be<br>
DISALLOWED. But I also don't want to go any further down the<br>
path toward picking out particular characters for special<br>
treatment than we have to. I'm more comfortable doing that if<br>
we can find a rule or if it is clear that the granularity (or<br>
some other aspect) of Unicode properties are not sufficient for<br>
the combination of the particular character and IDN<br>
applicability.<br>
<br>
I also have a vague recollection of a discussion about putting<br>
all of Lm into "contextual rule required", assigning characters<br>
with that property value rules only if there was a substantial<br>
reason for permitting them. We didn't do that for, if I<br>
recall, several reasons, but we did classify some of those<br>
characters that way by exception. I no longer understand some<br>
of those exceptions and why they did not cause others, which, of<br>
course, is one of the reasons why rule are better than lists.<br>
For example, we made<br>
<br>
U+02B9 MODIFIER LETTER PRIME<br>
<br>
CONTEXTO (because of its effective use in historical Greek<br>
numerals according to Tables, Appendix A.6), but treat<br>
<br>
02BA;MODIFIER LETTER DOUBLE PRIME<br>
02BB;MODIFIER LETTER TURNED COMMA<br>
02BC;MODIFIER LETTER APOSTROPHE<br>
02BD;MODIFIER LETTER REVERSED COMMA<br>
02BE;MODIFIER LETTER RIGHT HALF RING<br>
02BF;MODIFIER LETTER LEFT HALF RING<br>
02C0;MODIFIER LETTER GLOTTAL STOP<br>
02C1;MODIFIER LETTER REVERSED GLOTTAL STOP<br>
<br>
As PVALID because Lm is generally acceptable, despite the<br>
observation that some of those characters are are quite<br>
problematic in terms of confusion with punctuation, etc.<br>
<br>
So, your suggestion about Tatweel raises two other questions for<br>
me:<br>
<br>
(1) Do we need to revisit the Lm decision, either placing all of<br>
those characters in DISALLOWED and making exceptions where<br>
needed or placing all of them into CONTEXTO and assigning rules<br>
to specify relevant context if and when that is shown to be<br>
necessary? Note that the latter would permit some migration<br>
later, just by adding rules, while the former would ban any<br>
character for which we do not identify the need for an<br>
exception. On the other hand, because CONTEXTO does not require<br>
the same lookup-time check as DISALLOWED, it is a much weaker<br>
check than what you have suggested above (and the ASIWG<br>
recommended, at least if I recall that recommendation).<br>
<br>
FWIW, after skimming through the list of characters with General<br>
Category Lm that are not mapped out by NFKC (i.e., those that<br>
would not appear in IDNA input or that would be DISALLOWED for<br>
other reasons), removing Lm from the LetterDigits rule of Tables<br>
Section 2.1 and then making exceptions as needed for particular<br>
characters seems to me to be more with our supposed "inclusion<br>
list" model than including all of them and then excluding some<br>
on a character-by-character basis.<br>
<br>
(2) If the conclusion from the above is that we should leave<br>
Lm's treatment in Tables Section 2.1 alone, would you recommend<br>
that we exceptionally make some or all of 02BA-02C1 DISALLOWED<br>
along with Tatweel (and why or why not)?<br>
<font color="#888888"><br>
john<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</font></blockquote></div><br>