Unicode 7.0.0, (combining) Hamza Above, and normalization for comparison

Vint Cerf vint at google.com
Wed Aug 6 16:43:49 CEST 2014 

actually i think neither of those is the point. The point here is that
there are two ways to encode the same character and none of the
normalization mechanisms transforms the representation into a single,
canonical encoding. The result is ambiguity for comparing one label to
another in the domain names, leading to the potential for phishing. The
principle is to try to avoid that outcome.

vint



On Wed, Aug 6, 2014 at 10:26 AM, Mark Davis ☕️ <mark at macchiato.com> wrote:

> > that is not the argument mark
>
> I'm not sure why it isn't the argument. When I say *X is confusable with
> Y*, I mean "X looks like Y but is encoded differently". That appears to
> be exactly the same as your "different encodings of what looks like the
> same thing".
>
> So I'll try translating P1 into your terms:
>
> P1: Any characters that are visually confusable with others should be
> excluded from domain names.
> =>
> P1: Any characters that are have different encoding but look the same as
> others should be excluded from domain names.
>
>
> Mark <https://google.com/+MarkDavis>
>
>  *— Il meglio è l’inimico del bene —*
>
>
> On Wed, Aug 6, 2014 at 6:22 AM, Vint Cerf <vint at google.com> wrote:
>
>> that is not the argument mark - it has to do with different encodings of
>> what looks like the same thing - that can be exploited by phishing, for
>> example.
>>
>>  v
>>
>>
>>
>> On Wed, Aug 6, 2014 at 9:20 AM, Mark Davis ☕️ <mark at macchiato.com> wrote:
>>
>>> The problem with excluding this one particular character is that it
>>> introduces a discrepancy with no real value. It's a bit like addressing
>>> global warming by requiring that Harley Davidsons with licence plate
>>> numbers divisible by 3 cannot use the US interstate. It might make some
>>> people feel like they are doing something good, but just complicates the
>>> rules with no measurable benefit.
>>>
>>> The principle that is being espoused is something like:
>>>
>>> P1: Any characters that are visually confusable with others should be
>>> excluded from domain names.
>>>
>>> But why apply that principle to this (rather rare) character, that is
>>> only used in a particular language, when it is not applied to thousands of
>>> other characters, and characters that are vastly more commonly used? And if
>>> P1 is not the principle that is being applied, what is?
>>>
>>>
>>>
>>> Mark <https://google.com/+MarkDavis>
>>>
>>>  *— Il meglio è l’inimico del bene —*
>>>
>>>
>>> On Wed, Aug 6, 2014 at 5:02 AM, Vint Cerf <vint at google.com> wrote:
>>>
>>>> Mark,
>>>>
>>>> I think it is important to distinguish the use of a character for
>>>> purposes other than domain names and for use in domain names. The long
>>>> preface in your email makes that case that the character is useful and
>>>> legitimate in the language but I think it has long been understood that
>>>> domain names have properties and requirements for unambiguous comparison
>>>> that might lead to the conclusion that, despite the character's undeniable
>>>> utility in written language, it may still not be appropriate for use in
>>>> domain names.
>>>>
>>>> In consequence of that line of reasoning, I would separate the
>>>> legitimate presence of the character in UNICODE from its use in domain
>>>> names.
>>>>
>>>> The next question, of course, is whether there is any harm to users
>>>> that can be anticipated by the use of the character in domain names. I
>>>> think it is clear that whatever conclusion is reached, it should apply to
>>>> all levels of domain name and therefore to all labels. Adopting a rule
>>>> concerning a character that is applied only at TLD or SLD level but cannot
>>>> be enforced at lower levels of a DNS hierarchy seems like a mistake, so if
>>>> there is a problem with a character, that problem should be solved for all
>>>> use in labels.
>>>>
>>>> I want to emphasize that, so far in this text, I have not taken a
>>>> position regarding the use of U+08A1 in domain names. I am only discussing
>>>> the process of deciding whether a character should be rendered invalid for
>>>> use in domain name labels. The argument that the character is useful for
>>>> properly formed written language is not necessarily an argument for its
>>>> permitted use in domain names, if harms are identified in the latter usage
>>>> that are considered unacceptable.
>>>>
>>>> It is also clear from past experience that reasonable people can differ
>>>> in their assessment of the degree of risk or harm that a particular
>>>> character poses. So, now we get to the central question whether U+08A1
>>>> produces sufficient risk to be banned from use in domain names.
>>>>
>>>> In Klensin's draft RFC, the argument is made that the previous way in
>>>> which BEH WITH HAMZA ABOVE was accommodated, including for use in domain
>>>> names, was a sequence U+0628 followed by U+0654. The incorporation of
>>>> a new, combined character U+08A1 creates ambiguity because there would be
>>>> two ways for this character to be used in a domain name, but the two do not
>>>> compare equal. For users, this creates the risk that two labels could be
>>>> registered that would look the same but presumably take one to distinct
>>>> destinations in the event that two different registrations of labels
>>>> containing instances of these characters (pre-composed U+08A1 and the
>>>> U+0628/U+0654 sequence) were permitted. Making U+08A1 PVALID while also
>>>> allowing the earlier composed form creates exactly this ambiguity.
>>>>
>>>> This is not a trivial problem. There are similar problems even in the
>>>> purely Latin character cases where numbers and letters can look the same,
>>>> depending on fonts, but be distinct for purposes of label comparisons. The
>>>> fact that such problematic forms exist should not be an argument for
>>>> introducing more of them.
>>>>
>>>> I think this discussion boils down to a principle of not introducing
>>>> additional ambiguity where there had been none before. It is also fair to
>>>> say that this is not a question of excluding the character itself from
>>>> UNICODE for which ample argument has been made for its inclusion, but a
>>>> question of allowing or disallowing its use in domain name labels.
>>>>
>>>> vint
>>>>
>>>>
>>>>
>>>> On Tue, Aug 5, 2014 at 7:07 PM, Mark Davis ☕️ <mark at macchiato.com>
>>>> wrote:
>>>>
>>>>>
>>>>>  ​I hadn't heard back from John, but I'm guessing that the right
>>>>> place to discuss this is here​, based on Marc's email.
>>>>>
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Mark Davis ☕️ <mark at macchiato.com>
>>>>> Date: Wed, Jul 30, 2014 at 7:38 AM
>>>>> Subject: Re: Unicode 7.0.0, (combining) Hamza Above, and normalization
>>>>> for comparison
>>>>> To: John C Klensin <john+w3c at jck.com>, Patrik Fältström <
>>>>> paf at frobbit.se>
>>>>> Cc: member-i18n-core at w3.org, Asmus Freytag <asmusf at ix.netcom.com>
>>>>>
>>>>>
>>>>> ​On what email address is this being discussed?  I'd like to convey to
>>>>> that list some comments from an internal discussion​ about draft-
>>>>> klensin-idna-5892upd-unicode70-00.txt.
>>>>>
>>>>> (These are not my wording, but I agree with them. I edited slightly
>>>>> for flow. I will add that from a confusability standpoint, the proposed
>>>>> draft accomplishes nothing, since there are thousands of cases of
>>>>> confusable characters; restricting just this one character has no useful
>>>>> effect; like removing a quart of water from a lake.)
>>>>>
>>>>> For U+08A1, this certainly is a *letter* of Fula (Fulfulde, Pula, ...),
>>>>> a large language spoken across swaths of
>>>>> West Africa. Fula is mostly written with the Latin script,
>>>>> but Islamists also write it in Ajami (Arabic extensions for African
>>>>> languages), particularly in Guinea.
>>>>> See:
>>>>>
>>>>> http://en.wikipedia.org/wiki/Fula_orthographies
>>>>>
>>>>> The *letter* in question is the one used to write the phoneme /ɓ/,
>>>>> the bilabial implosive. See:
>>>>>
>>>>> http://en.wikipedia.org/wiki/%C6%81
>>>>>
>>>>> for the African alphabet convention for the Latin writing of this
>>>>> letter.
>>>>>
>>>>> For the Arabic Ajami alphabets for Fula, the form has been missing.
>>>>> For whatever reason, in at least one Fulfulde Ajami orthography,
>>>>> this implosive was (reasonably) represented by using a Hamza
>>>>> diacritic on the beh letter. Following the way such *diacritic* (ijam)
>>>>> letter derivations are encoded in the Unicode Standard, a separate,
>>>>> non-decomposed entry was required. Note that this use of Hamza
>>>>> is *different* from the Arabic (language) use of a combining Hamza
>>>>> to indicate a glottal stop, often in combination with a letter that
>>>>> is actually pronounced as a vowel.
>>>>>
>>>>> As to *why* it was encoded as a single, undecomposed letter,
>>>>> that is explained at length in the proposal document, as well
>>>>> as in the section on Hamza in the Unicode Standard, which you
>>>>> have referred to in the Internet Draft you mention.
>>>>>
>>>>> The newly encoded character U+08A1 for Unicode 7.0 has
>>>>> *already* been added to the relevant table "Arabic Letters
>>>>> With Hamza Above" in the draft core specification for
>>>>> Unicode 7.0, where, like the long-encoded U+0681
>>>>> and U+076C, it is noted as having no decomposition.
>>>>> (The core specification will be posted around October -- it
>>>>> is still undergoing its final editing for all the 7.0 additions.)
>>>>>
>>>>> U+08A1 does not have a canonical decomposition in Unicode 7.0
>>>>> (nor, of course, will it *ever* have a canonical decomposition,
>>>>> because of normalization stability). This is exactly the same treatment
>>>>> that U+0681 and U+076C got, and for exactly the same reasons.
>>>>> (And, as you know, of course, those characters date back to
>>>>> Unicode 4.1 for U+076C and even earlier, Unicode 1.1 for U+0681.)
>>>>>
>>>>> Note that it is incorrect to assert that U+08A1 ARABIC LETTER BEH
>>>>> WITH HAMZA ABOVE "should" be normalized to U+0628 ARABIC LETTER
>>>>> BEH + U+0654 ARABIC HAMZA ABOVE. Those are distinct sequences,
>>>>> and they are never going to compare equal in their NFC normalizations.
>>>>>
>>>>> I am concerned that the Internet Draft here is heading in
>>>>> exactly the wrong direction. If it ends up changing RFC 5892
>>>>> to override the derivation for U+08A1 and force it to INVALID,
>>>>> all I can see that accomplishing is to guarantee forever that
>>>>> correctly spelled Ajami Fulfulde cannot be used in domain
>>>>> names, and that instead people would end up having to use
>>>>> misspellings to represent their implosive b in a domain names.
>>>>>
>>>>> With all due respect to the Arabic script experts that have been
>>>>> consulted, I rather doubt that they are experts on Ajami orthographies
>>>>> in West Africa, or are in touch with the people who would be
>>>>> supporting those languages and implementing keyboarding
>>>>> and such for West Africa.
>>>>>
>>>>> Also, I don't see any way you can justify the abrupt (and permanent)
>>>>> discontinuity that this would put in place between the treatment
>>>>> of U+08A1 for Fulfulde and U+076C for Ormuri or U+0681 for Pashto.
>>>>>
>>>>>
>>>>> If you are looking for a more analogous precedent I suggest, for
>>>>> example:
>>>>>
>>>>> U+2C65 LATIN SMALL LETTER A WITH STROKE
>>>>>
>>>>> That was added in Unicode 5.0, and nobody has ever had any problem
>>>>> with it being PVALID in IDNA. It only has limited use in a minor
>>>>> orthography,
>>>>> but what is the harm?
>>>>>
>>>>> Now, if you examine U+2C65, you could well claim that it *should*
>>>>> be decomposed to "a" plus the combining stroke overlay, U+0338.
>>>>> And both of those have been encoded for a long, long time in
>>>>> the standard, so in principle, somebody *could* have been representing
>>>>> their data for a letter a with stroke before Unicode 5.0 using the
>>>>> sequence with the stroke
>>>>> overlay. It might even look o.k. in text, depending on the font support
>>>>> for the combina̸tion. But the Unicode Standard has rules now for the
>>>>> encoding of certain combinations of base letters and diacritic
>>>>> modifiers
>>>>> that overlay or modify the base character form. So U+2C65 was
>>>>> separately encoded. And there is no normalization of the sequence
>>>>> involved. That stroked letter use is, in text, distinct from somebody,
>>>>> say,
>>>>> using a bunch of overlay strokes as a strikethrough convention for
>>>>> some reason: a̸a̸a̸a̸a̸a̸
>>>>>
>>>>> Consider the Hamza diacritic as falling in this same class of edge
>>>>> cases,
>>>>> if you will.
>>>>>
>>>>> And in this case, I don't think it will be doing anybody any favors to
>>>>> update RFC 5892 to make U+08A1 DISALLOWED in IDNA. It doesn't
>>>>> "fix" normalization for it. All it accomplishes is to force any
>>>>> Fulfulde
>>>>> user of Ajami orthography to misspell their text in order to use a /ɓ/
>>>>> in a domain name. It would just create an unexplained (and unfixable)
>>>>> discontinuity between what the domain registrations would accept
>>>>> and what the Fulfulde input and spelling tools would support. Or I
>>>>> guess it would just force people to give up the Arabic spellings and
>>>>> go back to the more widely supported Latin alphabets for Fula to
>>>>> get their domain names.
>>>>>
>>>>> What would be accomplished by
>>>>> forcing another point incompatibility that just ends up getting
>>>>> carried around forever?
>>>>>
>>>>> ​====
>>>>> ​
>>>>> There are four levels at which confusables, including homoglyphs
>>>>> ​,​
>>>>> can be addressed for domain names
>>>>>
>>>>> 1. Encoding
>>>>> 2. Protocol (IDNA)
>>>>> 3. Label Generation Ruleset
>>>>> 4. String Review
>>>>>
>>>>> ​A
>>>>>  more natural level ​[for ​addressing ​confusables​] ​​would be the
>>>>> Label Generation Ruleset level. For an LGR, there are three ways to deal
>>>>> with homoglyphs, one of which is not available on the protocol level. The
>>>>> first two of these are to rule out a code point (by not including it in the
>>>>> LGR's repertoire), or to rule out a code point or sequence conditionally.
>>>>> Unlike using these methods on the Protocol level, doing so on the LGR level
>>>>> means that it is possible to be more restrictive, say, for the root of the
>>>>> DNS than for domains several levels down the tree. The downside of using
>>>>> the LGR is, of course, that it is specific to the given zone on the
>>>>> internet.
>>>>>
>>>>> The upside is that an LGR has additional mechanisms, such as defining
>>>>> a "blocked" variant. That creates an "either/or" situation, where both are
>>>>> permitted, but not at the same time in the same position of an otherwise
>>>>> identical label. This is a very nice solution for a number of
>>>>> confusables/homoglyphs that are systemic (not dependent on accidents of
>>>>> rendering or "arms length" similarity).
>>>>>
>>>>> Unlike the final level, String Review, an LGR has the advantage of
>>>>> being applied mechanically without any case-by-case review, which is why
>>>>> it's appropriate for cases like the one that gave rise to this discussion.
>>>>>
>>>>> In principle, both the Label Generation Ruleset or the String Review
>>>>> are created/carried out by people/entities that have access to the
>>>>> necessary and specific linguistic and script expertise, unlike IDNA which
>>>>> seems to be created largely by protocol experts.
>>>>>
>>>>>
>>>>> On Tue, Jul 22, 2014 at 12:22 AM, John C Klensin <john+w3c at jck.com>
>>>>> wrote:
>>>>>
>>>>>> Hi.   I was asked to forward the announcement of this Internet
>>>>>> Draft to this group once it was posted.  See attached.
>>>>>>
>>>>>> For information -- comments welcome, but the core issue may be
>>>>>> rather specific to concerns that surround IDNs and IDNA.  Or not.
>>>>>>
>>>>>> Or course, if I/we are still completely confused, corrections
>>>>>> and explanations would be welcome.
>>>>>>
>>>>>>     john
>>>>>>
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: internet-drafts at ietf.org
>>>>>> To: i-d-announce at ietf.org
>>>>>> Cc:
>>>>>> Date: Mon, 21 Jul 2014 04:03:58 -0700
>>>>>> Subject: I-D Action: draft-klensin-idna-5892upd-unicode70-00.txt
>>>>>>
>>>>>> A New Internet-Draft is available from the on-line Internet-Drafts
>>>>>> directories.
>>>>>>
>>>>>>
>>>>>>         Title           : IDNA Update for Unicode 7.0.0
>>>>>>         Authors         : John C Klensin
>>>>>>                           Patrik Faltstrom
>>>>>>         Filename        : draft-klensin-idna-5892upd-unicode70-00.txt
>>>>>>         Pages           : 10
>>>>>>         Date            : 2014-07-21
>>>>>>
>>>>>> Abstract:
>>>>>>    The current version of the IDNA specifications anticipated that
>>>>>> each
>>>>>>    new version of Unicode would be reviewed to verify that no changes
>>>>>>    had been introduced that required adjustments to the set of rules
>>>>>>    and, in particular, whether new exceptions or backward
>>>>>> compatibility
>>>>>>    adjustments were needed.  That review was conducted for Unicode
>>>>>> 7.0.0
>>>>>>    and identified a problematic new code point.  This specification
>>>>>>    updates RFC 5982 to disallow that code point and provides
>>>>>> information
>>>>>>    about the reasons why that exclusion is appropriate.  It also
>>>>>> applies
>>>>>>    an editorial clarification that was the subject of an earlier
>>>>>>    erratum.
>>>>>>
>>>>>>
>>>>>> The IETF datatracker status page for this draft is:
>>>>>> https://datatracker.ietf.org/doc/draft-klensin-idna-5892upd-unicode70/
>>>>>>
>>>>>> There's also a htmlized version available at:
>>>>>> http://tools.ietf.org/html/draft-klensin-idna-5892upd-unicode70-00
>>>>>>
>>>>>>
>>>>>> Please note that it may take a couple of minutes from the time of
>>>>>> submission
>>>>>> until the htmlized version and diff are available at tools.ietf.org.
>>>>>>
>>>>>> Internet-Drafts are also available by anonymous FTP at:
>>>>>> ftp://ftp.ietf.org/internet-drafts/
>>>>>>
>>>>>> _______________________________________________
>>>>>> I-D-Announce mailing list
>>>>>> I-D-Announce at ietf.org
>>>>>> https://www.ietf.org/mailman/listinfo/i-d-announce
>>>>>> Internet-Draft
>>>>>> <https://www.ietf.org/mailman/listinfo/i-d-announceInternet-Draft>
>>>>>> directories: http://www.ietf.org/shadow.html
>>>>>> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Idna-update mailing list
>>>>> Idna-update at alvestrand.no
>>>>> http://www.alvestrand.no/mailman/listinfo/idna-update
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.alvestrand.no/pipermail/idna-update/attachments/20140806/e4de6645/attachment-0001.html>

More information about the Idna-update mailing list