Unicode 7.0.0, (combining) Hamza Above, and normalization for comparison
vint at google.com
Wed Aug 6 14:02:14 CEST 2014
I think it is important to distinguish the use of a character for purposes
other than domain names and for use in domain names. The long preface in
your email makes that case that the character is useful and legitimate in
the language but I think it has long been understood that domain names have
properties and requirements for unambiguous comparison that might lead to
the conclusion that, despite the character's undeniable utility in written
language, it may still not be appropriate for use in domain names.
In consequence of that line of reasoning, I would separate the legitimate
presence of the character in UNICODE from its use in domain names.
The next question, of course, is whether there is any harm to users that
can be anticipated by the use of the character in domain names. I think it
is clear that whatever conclusion is reached, it should apply to all levels
of domain name and therefore to all labels. Adopting a rule concerning a
character that is applied only at TLD or SLD level but cannot be enforced
at lower levels of a DNS hierarchy seems like a mistake, so if there is a
problem with a character, that problem should be solved for all use in
I want to emphasize that, so far in this text, I have not taken a position
regarding the use of U+08A1 in domain names. I am only discussing the
process of deciding whether a character should be rendered invalid for use
in domain name labels. The argument that the character is useful for
properly formed written language is not necessarily an argument for its
permitted use in domain names, if harms are identified in the latter usage
that are considered unacceptable.
It is also clear from past experience that reasonable people can differ in
their assessment of the degree of risk or harm that a particular character
poses. So, now we get to the central question whether U+08A1 produces
sufficient risk to be banned from use in domain names.
In Klensin's draft RFC, the argument is made that the previous way in which
BEH WITH HAMZA ABOVE was accommodated, including for use in domain names,
was a sequence U+0628 followed by U+0654. The incorporation of a new,
combined character U+08A1 creates ambiguity because there would be two ways
for this character to be used in a domain name, but the two do not compare
equal. For users, this creates the risk that two labels could be registered
that would look the same but presumably take one to distinct destinations
in the event that two different registrations of labels containing
instances of these characters (pre-composed U+08A1 and the U+0628/U+0654
sequence) were permitted. Making U+08A1 PVALID while also allowing the
earlier composed form creates exactly this ambiguity.
This is not a trivial problem. There are similar problems even in the
purely Latin character cases where numbers and letters can look the same,
depending on fonts, but be distinct for purposes of label comparisons. The
fact that such problematic forms exist should not be an argument for
introducing more of them.
I think this discussion boils down to a principle of not introducing
additional ambiguity where there had been none before. It is also fair to
say that this is not a question of excluding the character itself from
UNICODE for which ample argument has been made for its inclusion, but a
question of allowing or disallowing its use in domain name labels.
On Tue, Aug 5, 2014 at 7:07 PM, Mark Davis ☕️ <mark at macchiato.com> wrote:
> I hadn't heard back from John, but I'm guessing that the right place to
> discuss this is here, based on Marc's email.
> ---------- Forwarded message ----------
> From: Mark Davis ☕️ <mark at macchiato.com>
> Date: Wed, Jul 30, 2014 at 7:38 AM
> Subject: Re: Unicode 7.0.0, (combining) Hamza Above, and normalization for
> To: John C Klensin <john+w3c at jck.com>, Patrik Fältström <paf at frobbit.se>
> Cc: member-i18n-core at w3.org, Asmus Freytag <asmusf at ix.netcom.com>
> On what email address is this being discussed? I'd like to convey to
> that list some comments from an internal discussion about draft-klensin
> (These are not my wording, but I agree with them. I edited slightly for
> flow. I will add that from a confusability standpoint, the proposed draft
> accomplishes nothing, since there are thousands of cases of confusable
> characters; restricting just this one character has no useful effect; like
> removing a quart of water from a lake.)
> For U+08A1, this certainly is a *letter* of Fula (Fulfulde, Pula, ...),
> a large language spoken across swaths of
> West Africa. Fula is mostly written with the Latin script,
> but Islamists also write it in Ajami (Arabic extensions for African
> languages), particularly in Guinea.
> The *letter* in question is the one used to write the phoneme /ɓ/,
> the bilabial implosive. See:
> for the African alphabet convention for the Latin writing of this letter.
> For the Arabic Ajami alphabets for Fula, the form has been missing.
> For whatever reason, in at least one Fulfulde Ajami orthography,
> this implosive was (reasonably) represented by using a Hamza
> diacritic on the beh letter. Following the way such *diacritic* (ijam)
> letter derivations are encoded in the Unicode Standard, a separate,
> non-decomposed entry was required. Note that this use of Hamza
> is *different* from the Arabic (language) use of a combining Hamza
> to indicate a glottal stop, often in combination with a letter that
> is actually pronounced as a vowel.
> As to *why* it was encoded as a single, undecomposed letter,
> that is explained at length in the proposal document, as well
> as in the section on Hamza in the Unicode Standard, which you
> have referred to in the Internet Draft you mention.
> The newly encoded character U+08A1 for Unicode 7.0 has
> *already* been added to the relevant table "Arabic Letters
> With Hamza Above" in the draft core specification for
> Unicode 7.0, where, like the long-encoded U+0681
> and U+076C, it is noted as having no decomposition.
> (The core specification will be posted around October -- it
> is still undergoing its final editing for all the 7.0 additions.)
> U+08A1 does not have a canonical decomposition in Unicode 7.0
> (nor, of course, will it *ever* have a canonical decomposition,
> because of normalization stability). This is exactly the same treatment
> that U+0681 and U+076C got, and for exactly the same reasons.
> (And, as you know, of course, those characters date back to
> Unicode 4.1 for U+076C and even earlier, Unicode 1.1 for U+0681.)
> Note that it is incorrect to assert that U+08A1 ARABIC LETTER BEH
> WITH HAMZA ABOVE "should" be normalized to U+0628 ARABIC LETTER
> BEH + U+0654 ARABIC HAMZA ABOVE. Those are distinct sequences,
> and they are never going to compare equal in their NFC normalizations.
> I am concerned that the Internet Draft here is heading in
> exactly the wrong direction. If it ends up changing RFC 5892
> to override the derivation for U+08A1 and force it to INVALID,
> all I can see that accomplishing is to guarantee forever that
> correctly spelled Ajami Fulfulde cannot be used in domain
> names, and that instead people would end up having to use
> misspellings to represent their implosive b in a domain names.
> With all due respect to the Arabic script experts that have been
> consulted, I rather doubt that they are experts on Ajami orthographies
> in West Africa, or are in touch with the people who would be
> supporting those languages and implementing keyboarding
> and such for West Africa.
> Also, I don't see any way you can justify the abrupt (and permanent)
> discontinuity that this would put in place between the treatment
> of U+08A1 for Fulfulde and U+076C for Ormuri or U+0681 for Pashto.
> If you are looking for a more analogous precedent I suggest, for example:
> U+2C65 LATIN SMALL LETTER A WITH STROKE
> That was added in Unicode 5.0, and nobody has ever had any problem
> with it being PVALID in IDNA. It only has limited use in a minor
> but what is the harm?
> Now, if you examine U+2C65, you could well claim that it *should*
> be decomposed to "a" plus the combining stroke overlay, U+0338.
> And both of those have been encoded for a long, long time in
> the standard, so in principle, somebody *could* have been representing
> their data for a letter a with stroke before Unicode 5.0 using the
> sequence with the stroke
> overlay. It might even look o.k. in text, depending on the font support
> for the combina̸tion. But the Unicode Standard has rules now for the
> encoding of certain combinations of base letters and diacritic modifiers
> that overlay or modify the base character form. So U+2C65 was
> separately encoded. And there is no normalization of the sequence
> involved. That stroked letter use is, in text, distinct from somebody, say,
> using a bunch of overlay strokes as a strikethrough convention for
> some reason: a̸a̸a̸a̸a̸a̸
> Consider the Hamza diacritic as falling in this same class of edge cases,
> if you will.
> And in this case, I don't think it will be doing anybody any favors to
> update RFC 5892 to make U+08A1 DISALLOWED in IDNA. It doesn't
> "fix" normalization for it. All it accomplishes is to force any Fulfulde
> user of Ajami orthography to misspell their text in order to use a /ɓ/
> in a domain name. It would just create an unexplained (and unfixable)
> discontinuity between what the domain registrations would accept
> and what the Fulfulde input and spelling tools would support. Or I
> guess it would just force people to give up the Arabic spellings and
> go back to the more widely supported Latin alphabets for Fula to
> get their domain names.
> What would be accomplished by
> forcing another point incompatibility that just ends up getting
> carried around forever?
> There are four levels at which confusables, including homoglyphs
> can be addressed for domain names
> 1. Encoding
> 2. Protocol (IDNA)
> 3. Label Generation Ruleset
> 4. String Review
> more natural level [for addressing confusables] would be the Label
> Generation Ruleset level. For an LGR, there are three ways to deal with
> homoglyphs, one of which is not available on the protocol level. The first
> two of these are to rule out a code point (by not including it in the LGR's
> repertoire), or to rule out a code point or sequence conditionally. Unlike
> using these methods on the Protocol level, doing so on the LGR level means
> that it is possible to be more restrictive, say, for the root of the DNS
> than for domains several levels down the tree. The downside of using the
> LGR is, of course, that it is specific to the given zone on the internet.
> The upside is that an LGR has additional mechanisms, such as defining a
> "blocked" variant. That creates an "either/or" situation, where both are
> permitted, but not at the same time in the same position of an otherwise
> identical label. This is a very nice solution for a number of
> confusables/homoglyphs that are systemic (not dependent on accidents of
> rendering or "arms length" similarity).
> Unlike the final level, String Review, an LGR has the advantage of being
> applied mechanically without any case-by-case review, which is why it's
> appropriate for cases like the one that gave rise to this discussion.
> In principle, both the Label Generation Ruleset or the String Review are
> created/carried out by people/entities that have access to the necessary
> and specific linguistic and script expertise, unlike IDNA which seems to be
> created largely by protocol experts.
> On Tue, Jul 22, 2014 at 12:22 AM, John C Klensin <john+w3c at jck.com> wrote:
>> Hi. I was asked to forward the announcement of this Internet
>> Draft to this group once it was posted. See attached.
>> For information -- comments welcome, but the core issue may be
>> rather specific to concerns that surround IDNs and IDNA. Or not.
>> Or course, if I/we are still completely confused, corrections
>> and explanations would be welcome.
>> ---------- Forwarded message ----------
>> From: internet-drafts at ietf.org
>> To: i-d-announce at ietf.org
>> Date: Mon, 21 Jul 2014 04:03:58 -0700
>> Subject: I-D Action: draft-klensin-idna-5892upd-unicode70-00.txt
>> A New Internet-Draft is available from the on-line Internet-Drafts
>> Title : IDNA Update for Unicode 7.0.0
>> Authors : John C Klensin
>> Patrik Faltstrom
>> Filename : draft-klensin-idna-5892upd-unicode70-00.txt
>> Pages : 10
>> Date : 2014-07-21
>> The current version of the IDNA specifications anticipated that each
>> new version of Unicode would be reviewed to verify that no changes
>> had been introduced that required adjustments to the set of rules
>> and, in particular, whether new exceptions or backward compatibility
>> adjustments were needed. That review was conducted for Unicode 7.0.0
>> and identified a problematic new code point. This specification
>> updates RFC 5982 to disallow that code point and provides information
>> about the reasons why that exclusion is appropriate. It also applies
>> an editorial clarification that was the subject of an earlier
>> The IETF datatracker status page for this draft is:
>> There's also a htmlized version available at:
>> Please note that it may take a couple of minutes from the time of
>> until the htmlized version and diff are available at tools.ietf.org.
>> Internet-Drafts are also available by anonymous FTP at:
>> I-D-Announce mailing list
>> I-D-Announce at ietf.org
>> directories: http://www.ietf.org/shadow.html
>> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
> Idna-update mailing list
> Idna-update at alvestrand.no
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Idna-update