1. I just did a test of the 05 tables (against Unicode 5.0).<br><br>Other than the Cf issue, I found one other thing. There are <reserved> characters (that is, General_Category=Cn) that show up as DISALLOWED when they shouldn't.<br>
<br>2064..2069 ; DISALLOWED # <reserved>..<reserved><br>...<br><br>I believe the reason is that they are default_ignoreable. But General_Category=Cn should take precedence.<br><br><br>2. In addition, this thread reinforces my opinion that the current use of lettered categories is needlessly obscure. Someone can't make heads nor tails of the rules unless they carefully decipher the cryptic Category A, B, ..., J, leafing back and forth in the document. As remarked before, there is no need to use FORTRAN-style single-letters for obscurity when we can use whole word labels for clarity.<br>
<br>That is, I think the rules should be something understandable. We can get the same results by casting the rules into the following form:<br><br><div style="margin-left: 40px;">CONTEXTJ = Join_Controls<br><br>CONTEXTO = Context_Exceptions<br>
<br>UNASSIGNED = Unassigned_Code_Points<br>
<br>
PVALID = Letters_Marks_Numbers<br> - Not_Stable_Under_NFKC_Case_Folding<br> - Default_Ignorables<br> - Block_Exceptions<br> - CONTEXTJ<br>
- CONTEXTO<br> + PValid_Exceptions<br> - Disallowed_Exceptions<br><br>DISALLOWED = <everything else><br></div>
<br>This gives the same results as the 05 formulation (with the exceptions given at the top: the unassigned 2064..2069,...), and is much more understandable. You can see at a glance where each category is making a contribution, instead of needing to follow a chain of logic. The necessary changes to the 05 text are not that large: <br>
<br>1. Renaming your categories:<br><br><div style="margin-left: 40px;">A => Letters_Marks_Numbers<br>B => Not_Stable_Under_NFKC_Case_Folding<br>C => Default_Ignorables<br>D => Block_Exceptions<br>E+F.1+G => PValid_Exceptions<br>
F.2 => Context_Exceptions<br>H => Join_Controls<br>I => (not necessary, as per discussion of Cf)<br>J => Unassigned_Code_Points<br></div><br>2. Making certain changes to the text in the category sections.<br><ul>
<li>Split F into two parts, depending on where the characters go.</li><li>Merge E, part of F and G into PValid_Exceptions. If you really wanted to keep E and G separate you could, as PValid_ASCII_Exceptions and PValid_Compatibility_Exceptions respectively.<br>
</li><li>Also remove noncharacters and whitespace from C. Since those can never be in Letters_Marks_Numbers anyway, they don't need to be subtracted.</li><li>Add a new section Disallowed_Exceptions. Currently empty, but could have exceptions (such as grandfathered characters) in the future.<br>
</li></ul>Mark<br><br><br><div class="gmail_quote">On Wed, Mar 19, 2008 at 10:55 AM, Paul Hoffman <<a href="mailto:phoffman@imc.org">phoffman@imc.org</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
In summary:<br>
<br>
a) The desired change was "Make sure everything in {Cf}, category I, is disallowed".<br>
<br>
b) We discovered that what was really wanted was "Make sure everything in {Cf}, category I, other than those with property Join_Control, category H, is disallowed". That leaves ZWNJ and ZWJ allowed with contextual rules.<br>
<br>
Patrik's first pass at the change did (a), not (b). That proposed change was:<br>
<br>
At 5:02 PM -0400 3/15/08, Patrik Fältström wrote:<br>
>OLD:<br>
<div class="Ih2E3d">> o If the codepoint is in Category H (Section 2.2.4), the value is<br>
> CONTEXTJ.<br>
> o If the codepoint is in Category I (Section 2.2.5), the value is<br>
> CONTEXTO.<br>
>> o If the codepoint is in Category B (Section 2.1.2), the value is<br>
> DISALLOWED.<br>
>> o If the codepoint is in Category C (Section 2.1.3), the value is<br>
> DISALLOWED.<br>
>> o If the codepoint is in Category D (Section 2.1.4), the value is<br>
> DISALLOWED.<br>
><br>
>NEW:<br>
> The algorithm to calculate the value of the derived property is as<br>
> follows.<br>
> > o If the codepoint is in Category B (Section 2.1.2), the value is<br>
> DISALLOWED.<br>
>> o If the codepoint is in Category C (Section 2.1.3), the value is<br>
> DISALLOWED.<br>
>> o If the codepoint is in Category D (Section 2.1.4), the value is<br>
> DISALLOWED.<br>
> o If the codepoint is in Category H (Section 2.2.4), the value is<br>
> CONTEXTJ.<br>
> o If the codepoint is in Category I (Section 2.2.5), the value is<br>
> CONTEXTO.<br>
<br>
</div>A different fix, one that achieves (b). would instead be:<br>
<br>
NEWER-YET:<br>
<div class="Ih2E3d"> The algorithm to calculate the value of the derived property is as<br>
follows.<br>
</div><div class="Ih2E3d"> o If the codepoint is in Category H (Section 2.2.4), the value is<br>
CONTEXTJ.<br>
o If the codepoint is in Category B (Section 2.1.2), the value is<br>
DISALLOWED.<br>
o If the codepoint is in Category C (Section 2.1.3), the value is<br>
DISALLOWED.<br>
o If the codepoint is in Category D (Section 2.1.4), the value is<br>
DISALLOWED.<br>
</div>That is, check H before B, C, and D. Also, note that we no longer need category I at all, because it is a subset of C. Thus, the entire category I can be removed.<br>
<br>
Can someone else check my work here?<br>
<div><div></div><div class="Wj3C7c">_______________________________________________<br>
Idna-update mailing list<br>
<a href="mailto:Idna-update@alvestrand.no">Idna-update@alvestrand.no</a><br>
<a href="http://www.alvestrand.no/mailman/listinfo/idna-update" target="_blank">http://www.alvestrand.no/mailman/listinfo/idna-update</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Mark