<br><br><div class="gmail_quote">On Mon, Mar 3, 2008 at 11:45 AM, Harald Tveit Alvestrand <<a href="mailto:harald@alvestrand.no">harald@alvestrand.no</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
--On 3. mars 2008 09:18 -0800 Mark Davis <<a href="mailto:mark.davis@icu-project.org">mark.davis@icu-project.org</a>> wrote:<br>
<br>
> Comments below.<br>
<br>
and responses inline....<br>
<div class="Ih2E3d"><br>
><br>
> On Sat, Mar 1, 2008 at 5:10 PM, Mark Davis <<a href="mailto:mark.davis@icu-project.org">mark.davis@icu-project.org</a>><br>
> wrote:<br>
>><br>
>> 1.2. Background and history<br>
><br>
> The history is useful, but shouldn't be right up at the front. The<br>
> majority of people reading this document won't care why it is the way it<br>
> is, they will care what the spec says and how to use it.<br>
><br>
> Sections 1.2, and all of 2, should be moved either to the Rationale<br>
> document with the rest of the rationale for all the other parts, or at<br>
> least moved to an appendix. Given the structure of the documents,<br>
> Rationale would be better.<br>
<br>
</div>Moving it out of the bidi document destroys the self-containedness of the<br>
document.<br>
My personal opinion is that anyone who wants to implement bidi SHOULD have<br>
to scan at least this much information at least once.<br>
<br>
So unless there's a strong push for this move, I'll not make this change.</blockquote><div><br>It is still self-contained, as a description of the specification. Neither of the other two component documents (protocol and tables) are self-contained regarding the rationale, so it is unclear why this has to be inconsistent with them.<br>
<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<div class="Ih2E3d"><br>
><br>
>><br>
>> The IDNA specification "Stringprep", [RFC3454] makes the following<br>
>> statement in its section 6 on the bidi algorithm, :<br>
>><br>
> ...<br>
><br>
>> The justification proposed is this:<br>
>><br>
>><br>
>> o No two labels, when presented in display order, should have the<br>
>> same sequence of characters without also having the same sequence<br>
>> of characters in network order. (This is the criterion that is<br>
>><br>
>> explicit in RFC 3454).<br>
><br>
> The above needs to be qualified, by adding something like "in the same<br>
> bidi context". That is, as pointed out below, if you change the embedding<br>
> context, 123-456 in one context may look like 456-123 in another; same<br>
> for abc.ABC and ABC.abc.<br>
<br>
</div>Erik, did you verify this within one context, or between contexts?</blockquote><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<div class="Ih2E3d"><br>
>><br>
>> o In a display of a string of labels, the characters of each label<br>
>> should remain grouped between the characters delimiting the<br>
>> labels.<br>
><br>
> You need to name both clauses, since you refer to them below. Something<br>
> like:<br>
><br>
> o Label Uniqueness Condition. No two labels, when presented in display<br>
> order, should have the<br>
> ...<br>
> o Label Grouping Condition. In a display of a string of labels, the<br>
> characters of each label<br>
<br>
</div>Either that, or "Condition A" and "Condition B".</blockquote><div><br>I really don't understand the point of obscuring the meaning of names by using single letters. Some leftover from FORTRAN I'm guessing ;-)<br>
<br>Why use single letters when a couple of words would make it clear to readers which condition is being discussed, and they don't constantly have to be looking up to see which condition is A and which is B.<br><br></div>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<div><div></div><div class="Wj3C7c">> ...<br>
><br>
>><br>
>> o These properties should hold true both when the string is embedded<br>
>><br>
>> in a paragraph with LTR direction and when it's embedded in a<br>
>> paragraph with RTL direction, as long as explicit directional<br>
>> controls are not used within the same paragraph.<br>
>><br>
>> Several stronger statements were considered and rejected, because<br>
>><br>
>> they seem to be impossible to fulfil within the constraints of the<br>
>> Unicode bidirectional algorithm. These include:<br>
>><br>
>> o The appearance of a label should be unaffected by its embedding<br>
>> context. This proved impossible even for ASCII labels; the label<br>
>><br>
>> "123-456" will have a different display order in an RTL context<br>
>> than in a LTR context.<br>
>><br>
>><br>
>><br>
>><br>
>> Alvestrand & Karp Expires August 17, 2008 [Page 7]<br>
>><br>
>> Internet-Draft IDNA RTL fix Feb 2008<br>
>><br>
>><br>
>><br>
>> o The sequence of labels should be consistent with network order.<br>
>> This proved impossible - a domain name consisting of the labels<br>
>> (in network order) L1.R1.R2.L2 will be displayed as L1.R2.R1.L2 in<br>
>><br>
>> an LTR context.<br>
>><br>
>> o The "remain grouped" property should remain true when directional<br>
>> controls (LRE, RLE, RLO, LRO, PDF) are used in the same paragraph<br>
>> (outside of the labels). Because these controls affect<br>
>><br>
>> presentation order in non-obvious ways, by affecting the "sor" and<br>
>> "eor" properties of the Unicode BIDI algorithm, the conditions<br>
>> above would be very hard to satisfy for an useful set of strings<br>
>><br>
>> if this was true. As long as these controls have no influence<br>
>> over the display of the domain name, no problem will be caused,<br>
>> but the exact criterion for "will not influence" is hard to<br>
>><br>
>> codify.<br>
><br>
> The above is too strong. We didn't actually attempt to see what<br>
> difference these would make. Certainly they are a different bidi context.<br>
<br>
</div></div>I did the tests. I'll stand behind the statement.<br>
If you have tests that say other things, present them, and we'll discuss.<br></blockquote><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<div class="Ih2E3d"><br>
> However, using overrides forces all characters to have the same<br>
> directionality, so it is actually *easy* to meet the above criteria in<br>
> that case, because having the same order will guarantee -- within *that*<br>
> context -- both disambiguation and contiguity.<br>
<br>
</div>You did not read the statement the way I intended it. Having an LRO on one<br>
side of the label and a PDF on the other side would indeed force a single<br>
context, but would also offer a certain way of creating confusable labels -<br>
I would regard that as being used "on" the labels.</blockquote><div><br>It's unclear exactly what you meant by your text, and exactly what you
are testing. Let me try to be clearer about what I'm saying. The ordering of strings in general will differ according
to the BIDI context; we're all clear on that. The question is what
those contexts are, and how they are being tested. The main ones are:<br>
<br>
C0. <L> ... <teststring> ...<br>
C1. <R> ... <teststring> ...<br>
C2. <LRE>... <teststring> ... <PDF><br>
C3. <RLE>... <teststring> ... <PDF><br>
C4. <LRO> ... <teststring> ... <PDF><br>
C5. <RLO> ... <teststring> ... <PDF><br>
<br>
I have not tested, but believe, that C0 = C2, and C1 = C3. If you have
a test that shows the contrary that would be useful to know, otherwise your text needs to be changed.<br><br>C4 and C5
represent different contexts, since they completely override
directionality. The Grouping condition should still be true, but
Uniqueness (across contexts) will not be, since in C4 we'd get "abc"
and in C5 we'd get "cba" (since it overrides Latin letters to have RTL
ordering). <br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<div class="Ih2E3d"><br>
<br>
><br>
> So I'd suggest just rewording to say that that wasn't a goal.<br>
<br>
</div>Nope. It was an actual found problem; I started out thinking it should be a<br>
goal, and found it impossible to attain.<br>
<div class="Ih2E3d"><br>
>> o The "no two labels display the same" should hold true between LTR<br>
>> paragraphs and RTL paragraphs. This was shown to be unsound.<br>
><br>
> The word "unsound" both here and a few lines down, is the wrong word. You<br>
> should either use "untenable" or "impossible" (as you had above).<br>
<br>
</div>"Untenable" is a good word.<br>
<div class="Ih2E3d">><br>
>><br>
>> o No two domain names should be displayed the same, even under<br>
>><br>
>> differing directionality. This was shown to be unsound, since the<br>
>> domain name (network) ABC.abc will have display order CBA.abc in<br>
>> an LTR context and abc.CBA in an RTL context, while the domain<br>
>><br>
>> name (network) abc.ABC will display as abc.CBA in an LTR context<br>
>> and as CBA.abc in an RTL context.<br>
>><br>
><br>
> ...<br>
><br>
>> The "remain grouped" property can be more formally stated as:<br>
>><br>
>> o Let "Delimiter chars" be a set of characters with the Unicode BIDI<br>
>><br>
>> properties CS, WS, ON. (These are commonly used to delimit labels<br>
>> - both the FULL STOP and the space are included.)<br>
>><br>
>> * ET, though it commonly occurs next to domain names in practice,<br>
>> is problematic: the context R CS L EN ET (for instance A.a1%)<br>
>><br>
>> makes the label L EN grow unstable.<br>
><br>
> "grow unstable" should be "become unstable". However, you only define<br>
> what "unstable" means below, so either you need to move the definition<br>
> up. I'd actually prefer being more explicit, since "unstable" can have<br>
> multiple meanings, and just use explicit phrasing like:<br>
><br>
> makes the label L EN break the Label Grouping condition.<br>
<br>
</div>That makes sense.<br>
<div class="Ih2E3d">><br>
>><br>
>> * ES commonly occurs in labels as HYPHEN-MINUS, but could also be<br>
>> used as a delimiter (for instance, the plus sign). It is left<br>
>> out here.<br>
>><br>
>><br>
>> o Let "Position" be the position of a character in a string (in<br>
>> network order)<br>
>><br>
>> o Let "Bidi position" be the position computed by the Unicode Bidi<br>
>> algorithm<br>
>><br>
>> In a paragraph with an embedded string formed from the substrings A B<br>
>><br>
>> L C D, where A and D are (possibly zero-length) legal labels, and B<br>
>> and C are single "Delimiter chars", the label L is a legal label if,<br>
>> for all A, B, C and D, the bidi position of all characters in L is<br>
>><br>
>> within the range of positions for the characters of L in the string,<br>
>> for both the LTR and RTL paragraph direction.<br>
><br>
> This doesn't make sense to me. "be within the range of positions for the<br>
> characters?"<br>
<br>
</div>I didn't find a more readable way of saying "they all stick together".</blockquote><div><br>The problem is that that isn't explicit enough, because the letters *don't* stick to adjacent letters: "ABcdEF" can be reordered so that B is not next to c. So it is easy for people to misunderstand that language. That is why I tried rephrased below.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<div class="Ih2E3d"><br>
><br>
> Moreover, you can't say that that makes a label *legal*, since it might<br>
> not be legal because of the other conditions in IDNA. The definition is<br>
> also circular, since you can only define L to be legal if you know what<br>
> makes A and B legal. And the discussion of a "paragraph" may be confusing<br>
> to people -- what does a paragraph have to do with a label condition?<br>
><br>
> I suggest something like:<br>
><br>
> A label L satisfies the Label Grouping Condition when for any Delimiter<br>
> Characters D1 and D2 and any other strings S1 and S2 (possibly of length<br>
> zero):<br>
><br>
><br>
> If the string formed by concatenating S1, D1, L, D2, S2 is subject to<br>
> bidi reordering,<br>
> then all of the characters of L2 in the reordered string are between D1<br>
> and D2.<br>
><br>
> - The bidi reordering of L1, D1, L, D2, L2 may result in D2 coming<br>
> before D1<br>
> - Because S1 is any string, the bidi algorithm may set the paragraph<br>
> direction for the string to either Right-To-Left or Left-To-Right; thus<br>
> the reordering condition has to work in both bidi contexts.<br>
<br>
</div>that indeed sounds better.<br>
<br>
S1 isn't allowed to be "any string". It's constrained to be a valid label.</blockquote><div><br>I had trouble with the wording for that, because if the S1 are constrained to be legal, then the definition is circular:<br>
<br>L is legal if .... where S1 and S2 are legal.<br><br>Not sure how to get around that. Perhaps someone else can take a shot.<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<div><div></div><div class="Wj3C7c"><br>
><br>
>><br>
>> (The "zero-length" case represents the case where a domain name is<br>
>> next to something that isn't a domain name, separated by a delimiter<br>
>><br>
>> character).<br>
>><br>
>> The "No two labels" property can be formally stated as:<br>
>><br>
>> If two labels L and L', embedded as for the test above, displayed in<br>
>> a paragraph with the same directionality, are rearranged into the<br>
><br>
> rearranged => bidi reordered<br>
>><br>
>><br>
>><br>
>><br>
>> Alvestrand & Karp Expires August 17, 2008 [Page 9]<br>
>><br>
>> Internet-Draft IDNA RTL fix Feb 2008<br>
>><br>
>><br>
>> same sequence of codepoints, neither L nor L' is a legal label.<br>
>><br>
>><br>
>><br>
>> 4. A replacement for the RFC 3454 criterion<br>
>><br>
>> A set of rules that satisfies the tests above is as follows. The<br>
>> main bullets give the rule, subordinate bullets (if any) give<br>
>> justifications or examples of things that break if this rule is not<br>
>><br>
>> present. The term "unstable" means that it fails to satisfy the<br>
>> "remain grouped" property defined above.<br>
><br>
> remove this, and change all instances to "fail the Label Grouping<br>
> condition".<br>
><br>
>><br>
>> Exhaustive testing has verified that strings that satisfy this<br>
>> criterion satisfy both the requirements above at least for all<br>
>><br>
>> strings up to 6 characters.<br>
><br>
> [[The above doesn't make it clear that these are in fact the bidi<br>
> requirements that are referred to by the IDNA protocol document. I<br>
> suggest the following rewording:]]<br>
><br>
> Based on exhaustive testing of strings, the following conditions on<br>
> characters have been developed to meet the Label Grouping and Label<br>
> Uniqueness conditions.<br>
><br>
> Bidi Label Requirement<br>
><br>
> A series of labels where any one of them has a character of type R, AL or<br>
> AN must meet all of the following conditions:<br>
><br>
> [[then number each one for reference]]<br>
><br>
>><br>
>> o Only characters with the BIDI properties L, R, AL, AN, EN, ES, BN,<br>
>> ON and NSM are allowed.<br>
>><br>
>> * B, S and WS are excluded because they are separators or spaces.<br>
>><br>
>><br>
>> * LRE, LRO, RLE, RLO, PDF are excluded because they are bidi<br>
>> controls.<br>
>><br>
>> * ET is excluded because the string L ET is unstable.<br>
>><br>
>> * CS is excluded because the string L CS is unstable.<br>
>><br>
>><br>
>> o ES and ON are not allowed in the first position<br>
>><br>
>> * ES R and ON R are both unstable.<br>
>><br>
>> o ES and ON, followed by zero or more NSM, is not allowed in the<br>
>> last position<br>
>><br>
>> * L ON and L ES are both unstable.<br>
>><br>
>><br>
>> o If an L is present, no R, AL or AN may be present, and vice versa.<br>
>><br>
>> o If an EN is present, no AN may be present, and vice versa.<br>
>><br>
>> o The first character may not be an NSM<br>
>><br>
>> o The first character may not be an EN (European Number) or an AN<br>
>><br>
>> (Arabic Number).<br>
>><br>
>> * If the character on both sides of a CS is an EN or an AN, the<br>
>> labels turn unstable.<br>
>><br>
>><br>
>><br>
>><br>
>> Alvestrand & Karp Expires August 17, 2008 [Page 10]<br>
>><br>
>><br>
>> Internet-Draft IDNA RTL fix Feb 2008<br>
>><br>
>><br>
>> * Some domain names where some of the labels use leading EN and<br>
>> AN may be problem-free, but there's no way of verifying this<br>
>><br>
>> while looking at a single label in isolation.<br>
>><br>
>> * NOTE: This is a restriction on ASCII labels when used together<br>
>> with IDNA labels. This is a change from the existing rules for<br>
>> ASCII labels.<br>
>><br>
>><br>
>> * We could achieve stability by barring numbers at the end of<br>
>> labels, but this may be more disruptive in practice.<br>
>><br>
>><br>
>> 5. Other issues in need of resolution<br>
>><br>
>> This document concerns itself only with the rules that are needed<br>
>><br>
>> when dealing with domain names with characters that have differing<br>
>> Bidi properties, and considers characters only in terms of their Bidi<br>
>> properties. All other issues with these scripts have to be<br>
>> considered in other contexts.<br>
>><br>
>><br>
>> Another set of issues concerns the proper display of IDNs with a<br>
>> mixture of LTR and RTL labels, or only RTL labels.<br>
>><br>
>> It is unrealistic to expect that domain names will be written using<br>
>> embedded formatting codes between their labels; thus, the display<br>
><br>
> According to IDNA2003, it is not only unrealistic but also illegal. Or<br>
> maybe you mean the pre-preprocessed form? This needs fixing, depending on<br>
> what you mean.<br>
<br>
</div></div>Some people have suggested that one should display domain names as<br>
<label><lrm>.<label>... or similar "interesting" constructs. I think that's<br>
unrealistic. I'm not sure it's illegal.</blockquote><div><br>Ah, I see what you are saying now. It is illegal to have a U-Label or A-Label with an <rlm> in it, which is why I said that it is illegal. What you mean is that it would be possible for a U-Label *to be transformed for display by inserting <rlm>*, but that it is unrealistic to expect that to be done. So if you clarify the text to clearly mean that, it'd be fine.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<div><div></div><div class="Wj3C7c"><br>
>><br>
>> order will be determined by the bidirectional algorithm. Thus, a<br>
>> sequence (in network order) of R1.R2.ltr will be displayed in the<br>
>> order 2R.1R.ltr in a LTR context, which might surprise someone<br>
>> expecting to see labels displayed in hierarchical order. Again, this<br>
>><br>
>> memo does not attempt to suggest a solution to this problem.<br>
>><br>
>><br>
>> 6. Compatibility considerations<br>
>><br>
>> 6.1. Backwards compatibility considerations<br>
>><br>
>> As with any change to an existing standard, it is important to<br>
>><br>
>> consider what happens with existing implementations when the change<br>
>> is introduced. The following troublesome cases have been noted:<br>
>><br>
>> o Old program used to input the newly allowed string. If the old<br>
>><br>
>> program checks the input against RFC 3454, the string will not be<br>
>> allowed, and that domain name will remain inaccessible.<br>
>><br>
>> o Old program is asked to display the newly allowed string, and<br>
>> checks it against RFC 3454 before displaying. The program will<br>
>><br>
>> perform some kind of fallback, most likely displaying the Punycode<br>
>> form of the string.<br>
>><br>
>><br>
>><br>
>> Alvestrand & Karp Expires August 17, 2008 [Page 11]<br>
>><br>
>> Internet-Draft IDNA RTL fix Feb 2008<br>
>><br>
>><br>
>><br>
>> o Old program tries to display the newly allowed string. If the old<br>
>> program has code for displaying the last character of a string<br>
>> that is different from the code used to display the characters in<br>
>><br>
>> the middle of the string, display may be inconsistent and cause<br>
>> confusion.<br>
><br>
> I don't understand this. Why would anyone want to do this? While would<br>
> they pick the last character, or the first, or the 3rd from the last, or<br>
> any particular other character and assume they can tell the<br>
> directionality. Why call this out in particular?<br>
<br>
</div></div>I don't know why anyone would want to do this. What I know is that I can't<br>
guarantee that nobody's done this in existing code, the old programs (if<br>
any) are already out there, and neither you nor I can get at them to fix<br>
them.<br>
<br>
It's been raised as an issue, so I'll include it. (Same answer as last time<br>
you asked.)</blockquote><div><br><br>I just don't understand why you are calling out that as an issue, over a thousand equally incorrect assumptions. Is it only because someone raised it? Who raised it? If I raise the issue that if a program tried to assume
the directionality from the 3rd to the last character, then you'll
include that as an issue?<br>
<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<br>
<br>
Thanks for the comments. I'll incorporate some of them into -05.</blockquote><div><br>Again, I think this document is progressing quite nicely; with some more work to clarify the text we should be in good shape. The one thing I can't quite see clearly yet is the best way to phrase the Grouping condition; that's a bit tricky.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
<font color="#888888"><br>
Harald<br>
<br>
</font></blockquote></div><br><br clear="all"><br>-- <br>Mark