What I was trying to do is set expectations correctly. That is:<br><ul><li>It is false to say that the contextual rules will disallow joiners in all cases where they make no visual difference.</li><li>It is true to say that the contextual rules can remove the vast majority of cases where joiners make no visual difference.</li>
</ul>As to the issue you cite, let me give a bit of background. The joiners were originally designed to be purely formatting controls, to indicate display preferences. More precisely, they are requests for rendering a particular way, ideally to be honored by the font/rendering system if possible. But a font/rendering system may not be able to honor the request, and would then just give the normal display.<br>
<br>The goal was indeed to encode characters where a <i>semantic</i> difference (rather than a display preference) was necessary. Thus these characters were meant to be optional. However, semantic vs display differences are to a certain degree a judgment call, and communities have developed with the expectation of behavior whereby the presence or absence of the joiner would be seen as the "wrong" spelling of the word.<br>
<br>Now, in the case of Malayalam, the chilu characters were added because it was not possible to represent certain text with the current characters. And that is a good reason for adding new characters. In so doing, for the purpose of identifiers, it is the case that the joiners are not really necessary. And while the UTC has no stated opinion on this issue, it would certainly consider proposals for the addition of characters to represent semantic differences for Indic scripts that may currently be using joiner/non-joiner.<br>
<br>Note -- and this is important -- that the addition of characters that have the same effect as "virama + non-joiner" or "virama + joiner" often do absolutely nothing to reduce cases of visual similarity: the new characters will look just like other characters in circumstances where the joiners had no effect! The advantage from the IDNA perspective, however, is that it makes testing easier.<br>
<br>I am less familiar with the case of Sinhala, but I believe that the main issue with all of the other Indic-based scripts is the same as the sigma case. That is, it is not a significant problem to identify input strings A and B; the problem is that the one that is returned from the DNS is not the "correct" form. That is, the canonicalization and equivalence class are acceptable, but the particular representative of that equivalence class that is in the DNS is not the preferred form. For some cases, as with Arabic, two words with different meanings may map together, but frankly, that is on the order of the difference between:<br>
<br><a href="http://therapist.com">therapist.com</a> // read as "Therapist"<br><a href="http://therapist.com">therapist.com</a> // read as "the rapist"<br><br>Personally, I believe that we could get along quite without having the four special cases (eszett, sigma, joiners) in IDNA2008. That is, the advantages of compatibility outweigh the utility of a breaking change. And as we all know, IDNA does not guarantee that all the text in any given language can be in IDNs: the simplest example is the English word: <b>can't</b>.<br>
<br>That is my personal opinion -- not necessarily representing particular Unicode Consortium members, who may have different, reasonable views of the importance of these cases.<br><br clear="all">Mark<br>
<br><br><div class="gmail_quote">On Fri, Feb 27, 2009 at 07:43, John C Klensin <span dir="ltr"><<a href="mailto:klensin@jck.com">klensin@jck.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
--On Friday, February 27, 2009 07:27 -0800 Mark Davis<br>
<<a href="mailto:mark@macchiato.com">mark@macchiato.com</a>> wrote:<br>
<br>
>...<br>
<div class="Ih2E3d">> With Indic scripts, the situation is slightly different. The<br>
> rules limit the cases severely, disallowing joiners where they<br>
> don't make a visual difference after almost all characters.<br>
> However, taking the example of Malayalam, something like half<br>
> of the cases where it allows joiners will not typically have a<br>
> difference in visual display. With Tamil even fewer, with<br>
> Sinhala, more.<br>
<br>
</div>FWIW, I was told early in this week that, with additional<br>
precomposed characters added in Unicode 5.1, Malayalam doesn't<br>
need joiners in IDNs at all. The person who made the comment<br>
more or less suggested that the proper solution to the issue<br>
with joiners and the other Indic scripts was to add sufficient<br>
additional characters to Unicode to capture the important<br>
shaping forms (e.g., the half-characters), thereby also<br>
eliminating the requirement. He felt that was a better solution<br>
generally because people who write the language understand about<br>
half-characters but have never heard of ZWJ.<br>
<br>
I did not encourage him to either make the relevant proposals or<br>
to sit around waiting for this to happen (proposals or not).<br>
<br>
However, to your specific point, Mark: do you suggest that the<br>
fact that we cannot completely cover all cases implies that we<br>
should give up and either abandon the joiners or leave this all<br>
to the registries?<br>
<font color="#888888"><br>
john<br>
<br>
</font></blockquote></div><br>