I wanted to clear up a few additional items, because I think the terminology is getting in the way.<br><br><b>1. unassigned</b><br>In Unicode, what we've been referring to as "unassigned" (more precisely gc=Cn) means that a code point (from 0 to 10FFFF) is not assigned *<i>to a character</i>*. The code point may actually have properties even though it does not represent a character: it might have bidi properties, block properties, or, as in this case, be default-ignoreable or a noncharacter. Sometimes a code point is called an "unassigned character" when what is meant is that it is a code point that is not assigned to be a character.<br>
<br><b>2. noncharacter</b><br>The term "noncharacter" is not the same as "unassigned character". Instead, they are a handful of special entities that can best be thought of as "super-private-use" code points, intended for internal use but not for interchange. They are and always will be unassigned (gc=Cn), but there are many (hundreds of thousands of) other gc=Cn that are not noncharacters.<br>
<br><b>2. the problem</b><br>The original email that sparked this whole discussion was my noting a problem "Other than the Cf issue, I found one other thing. There are
<reserved> characters (that is, General_Category=Cn) that show up
as DISALLOWED when they shouldn't.<br>
<br>
2064..2069 ; DISALLOWED # <reserved>..<reserved>"<br><br>I think this may have been misunderstood as meaning that these were the only code points that had this problem. That is not true, this is one of many other cases in tables-05. Another one, in particular, is:<br>
<br>FFFC..FFFF ; DISALLOWED # OBJECT REPLACEMENT CHARACTER..<reserved><br><br>Note that FFFF is a noncharacter.<br><br>The tag "<reserved>" is attached to code points that are not assigned, so any line in tables that contains DISALLOWED...<reserved> is a case where a Unicode unassigned character (gc=Cn) is being categorized as DISALLOWED.<br>
<br><b>4. default ignorables</b><br>Assigned default ignorable characters are are those are normally invisible if not supported, and thus should be DISALLOWED. There are also default ignorable <i>unassigned</i> characters, such as 2069 above. Those are in areas reserved for future Default Ignorables. (Default ignorables used to contain noncharacters, but don't in U5.1, for reasons explained earlier.)<br>
<br><br><br>So, where do we go from here? I'll try to set out the options.<br><br><div style="margin-left: 40px;"><i>A. First test for unassigned:</i><br><br></div><div style="margin-left: 80px;">gc=Cn => UNASSIGNED<br>
(default ignorable & gc!=Cn) => DISALLOWED<br></div>
<div style="margin-left: 40px;"><br>
Advantage: it means that Unicode unassigned = UNASSIGNED, which is conceptually simpler for people. Functionally, this works, because noncharacters will never change from UNASSIGNED and thus never be allowed in labels, and unassigned default ignorable characters will become DISALLOWED as soon as they are assigned.<br>
<br><br><i>B. First test for noncharacters, then unassigned</i><br><br></div><div style="margin-left: 80px;">noncharacters => DISALLOWED<br>(default ignorable & gc!=Cn) => DISALLOWED<br>(gc=Cn - noncharacters) => UNASSIGNED<br>
</div><div style="margin-left: 40px;"><br>Advantage: we know that noncharacters will never be PVALID so we might as well indicate that.<br><br><br><i>C. First test for noncharacters and default ignorable, then for unassigned</i><br>
<br></div><div style="margin-left: 80px;">noncharacters => DISALLOWED<br>(default ignorable) => DISALLOWED<br>
(gc=Cn - noncharacters - default_ignorable) => UNASSIGNED<br></div>
<div style="margin-left: 40px;"><br>
Advantage: we know that noncharacters and default ignorables will never be PVALID so we might as well indicate that.<br></div><br>I actually don't care to much which of these options we choose - functionally it doesn't make a difference. Here I'm just trying to present a clearer picture of the situation.<br>
<br>Mark<br><br><div class="gmail_quote">On Wed, Apr 30, 2008 at 5:35 PM, Kenneth Whistler <<a href="mailto:kenw@sybase.com">kenw@sybase.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Let me step in and take a crack at this. I think at this point<br>
you two are talking past each other and confusing everybody<br>
on the list.<br>
<div class="Ih2E3d"><br>
> At 1:38 PM -0700 4/30/08, Mark Davis wrote:<br>
> >It *is* related to Noncharacters. Default_Ignorable_Code_Point is a<br>
> >derived property. The code points that are unassigned (gc=Cn) but<br>
> >that should be DISALLOWED are all and only the Noncharacters.<br>
<br>
</div>What Mark is saying can be boiled down to the following<br>
observation. The draft-ietf-idna-tables-00.txt currently<br>
contains the following entry in the table:<br>
<div class="Ih2E3d"><br>
200E..2071 ; DISALLOWED # LEFT-TO-RIGHT MARK..SUPERSCRIPT LATIN SMALL<br>
<br>
</div>That is incorrect, because there are unassigned characters<br>
in that range. The correct entries in the table should be (for<br>
Unicode 5.1):<br>
<br>
200E..2064 ; DISALLOWED # LEFT-TO-RIGHT MARK..INVISIBLE PLUS<br>
2065..2069 ; UNASSIGNED # <reserved>..<reserved><br>
206A..2071 ; DISALLOWED # INHIBIT SYMMETRIC SWAPPING..SUPERSCRIPT LATI<br>
<br>
O.k. Are we all on board with that?<br>
<br>
If so, that means that there is either a bug in the statement<br>
of the various property-related classes or in the algorithm<br>
for the table derivation, or both.<br>
<div class="Ih2E3d"><br>
><br>
> Then I'm really confused. From the new draft:<br>
><br>
</div><div class="Ih2E3d">> <a href="http://2.1.3." target="_blank">2.1.3.</a> IgnorableProperties (C)<br>
><br>
> C: property(cp) is in {Default_Ignorable_Code_Point, White_Space,<br>
> Noncharacter_Code_Point}<br>
><br>
> This category is used to group codepoints that are not recommended<br>
> for use in identifiers. In general, these codepoints are not<br>
> suitable for use for IDN.<br>
><br>
> The definition for Default_Ignorable_Code_Point can be found in<br>
> DerivedCoreProperties.txt [1] (and erratum of 2007-January-25 [2])<br>
> and is<br>
><br>
> Other_Default_Ignorable_Code_Point + Cf + Cc + Cs<br>
> + Noncharacter_Code_Point + Variation_Selector<br>
> - White_Space - FFF9..FFFB (Annotation Characters)<br>
<br>
</div>O.k. next problem. Mark pointed out that 2.1.3 itself has not been<br>
correctly updated, because it still reflects a statement as of<br>
Unicode 5.0 (plus an erratum notice), instead of Unicode 5.1.<br>
<br>
The definition of Default_Ignorable_Code_Point for Unicode 5.0 is:<br>
<div class="Ih2E3d"><br>
# Derived Property: Default_Ignorable_Code_Point<br>
</div># Other_Default_Ignorable_Code_Point<br>
<div class="Ih2E3d"># + Cf<br>
# + Cc + Cs<br>
# + Noncharacter_Code_Point<br>
# + Variation_Selector<br>
# - White_Space<br>
# - FFF9..FFFB (Annotation Characters)<br>
<br>
</div>The definition of Default_Ignorable_Code_Point for Unicode 5.1 is:<br>
<div class="Ih2E3d"><br>
# Other_Default_Ignorable_Code_Point<br>
</div># + Cf (Format characters)<br>
<div class="Ih2E3d"># + Variation_Selector<br>
# - White_Space<br>
# - FFF9..FFFB (Annotation Characters)<br>
</div><div class="Ih2E3d"># - 0600..0603, 06DD, 070F (exceptional Cf characters that should be visible)<br>
<br>
</div>So, this means that the last two paragraphs of 2.1.3 need to<br>
be updated to provide the *correct* definition of<br>
Default_Ignorable_Code_Point.<br>
<br>
><br>
<div class="Ih2E3d">> Why have what whole list of things for "Default_Ignorable_Code_Point"<br>
> if all we want is Noncharacter_Code_Point, which is already in the<br>
> list for C? Why not have it at all?<br>
<br>
</div>First, I presume that is a typo for "Why have it at all?"<br>
<br>
Second, we aren't after *just* noncharacters. Those should certainly<br>
be disallowed, but Default_Ignorable_Code_Point picks up all<br>
the gc=Cf characters that should be DISALLOWED as well.<br>
<br>
So what is the problem here?<br>
<br>
The problem is an ordering problem in the application of rules<br>
for deriving the table. If you work through the actual derivation<br>
of Default_Ignorable_Code_Point, you get the results which<br>
are explicitly listed in DerivedCoreProperties.txt for Unicode 5.1:<br>
<br>
2060..2064 ; Default_Ignorable_Code_Point # Cf [5] WORD JOINER..INVISIBLE<br>
PLUS<br>
2065..2069 ; Default_Ignorable_Code_Point # Cn [5]<br>
<reserved-2065>..<reserved-2069><br>
206A..206F ; Default_Ignorable_Code_Point # Cf [6] INHIBIT SYMMETRIC<br>
SWAPPING..NOMINAL DIGIT SHAPES<br>
<br>
The entire range 2060..206F is Default_Ignorable_Code_Point=True,<br>
but, 2060..2064 and 206A..206F are assigned characters (and gc=Cf),<br>
whereas 2065..2069 are *NOT* assigned characters, and hence<br>
are gc=Cn.<br>
<br>
The intent of the table derivation for draft-ietf-idna-tables-00.txt,<br>
as I understand it is that all unassigned code points should<br>
always be UNASSIGNED in the table, regardless of what other<br>
properties they might have in Unicode data files.<br>
<br>
Therefore, we have an ordering bug in the algorithm that is<br>
deriving the table, because it has decided that 2065..2069<br>
should be DISALLOWED, based on their occurring in the<br>
class defined by category C, when they clearly should be<br>
UNASSIGNED, based on their status as unassigned in Unicode 5.1.<br>
<br>
O.k., that is complicated, I realize, but I hope that *now* it<br>
is clear what the problems are, at least.<br>
<br>
--Ken<br>
<div><div></div><div class="Wj3C7c"><br>
_______________________________________________<br>
Idna-update mailing list<br>
<a href="mailto:Idna-update@alvestrand.no">Idna-update@alvestrand.no</a><br>
<a href="http://www.alvestrand.no/mailman/listinfo/idna-update" target="_blank">http://www.alvestrand.no/mailman/listinfo/idna-update</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Mark