<font face="'times new roman', serif"><div style="background-color:transparent;margin-top:0px;margin-left:0px;margin-bottom:0px;margin-right:0px;font-size:medium"><span style="font-size:small"><br></span></div><div style="background-color:transparent;margin-top:0px;margin-left:0px;margin-bottom:0px;margin-right:0px;font-family:Times;font-size:medium">
<span style="font-family:'times new roman',serif;font-size:small">Mark</span></div></font><font face="'times new roman', serif"><i>— Il meglio è l’inimico del bene —</i></font><div><font face="'times new roman', serif"><i><br>
</i></font><div><font face="'times new roman', serif"><i><span style="font-style:normal"><div style="background-color:transparent;margin-top:0px;margin-left:0px;margin-bottom:0px;margin-right:0px">[<a href="https://plus.google.com/114199149796022210033" target="_blank">https://plus.google.com/114199149796022210033</a>]</div>
<i></i></span><i></i></i></font></div></div><br>
<br><br><div class="gmail_quote">On Mon, Jan 30, 2012 at 05:07, Gervase Markham <span dir="ltr"><<a href="mailto:gerv@mozilla.org">gerv@mozilla.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Mark,<br>
<br>
Again, thanks for your very helpful input.<div class="im"><br>
<br>
On 23/01/12 21:12, Mark Davis ☕ wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The Unicode Consortium in U6.1 (due out soon) is adding the property<br>
Script_Extensions, to provide that data. The sample code in #39 should<br>
be updated to include that, so handling those cases.<br>
</blockquote>
<br></div>
Can you be a bit more specific about "soon"? :-)<br></blockquote><div><br></div><font face="times new roman,serif">"soon" for 6.1 and UTS #46 is February</font></div><div class="gmail_quote"><font face="times new roman,serif"><br>
</font></div><div class="gmail_quote"><font face="times new roman,serif">There's a UTC meeting in a week that will be reviewing </font><span style="font-family:'times new roman',serif">TR36/39</span><font face="times new roman,serif"> (so any feedback people have on them is welcome.)</font></div>
<div class="gmail_quote"><font face="times new roman,serif"><br></font></div><div class="gmail_quote"><font face="times new roman,serif"><a href="http://www.unicode.org/reports/tr36/proposed.html">http://www.unicode.org/reports/tr36/proposed.html</a></font></div>
<div class="gmail_quote"><font face="times new roman,serif"><a href="http://www.unicode.org/reports/tr39/proposed.html">http://www.unicode.org/reports/tr39/proposed.html</a><br clear="all"></font><font face="'times new roman', serif"></font><div>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
So this data will associate a number (N, > 1) of language names with each Common or Inherited character?</blockquote><div><br></div><div>No, it associates multiple scripts with certain common/inherited characters; the data is at </div>
<div><br></div><div><a href="http://unicode.org/Public/6.1.0/ucd/ScriptExtensions.txt">http://unicode.org/Public/6.1.0/ucd/ScriptExtensions.txt</a></div><div><br></div><div>For example, the katakana/hiragana mark is Common, but the newer data allows people to detect it in (say) a Cyrillic string.</div>
<div><pre style="word-wrap:break-word;white-space:pre-wrap">30FC ; Hira Kana # Lm KATAKANA-HIRAGANA PROLONGED SOUND MARK</pre></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Most of the check for different numbering systems is handled by the<br>
script detection. The only real additional work is to verify there there<br>
is no more than one numbering system.<br>
</blockquote>
<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
* Check to see that all the characters are in the sets of exemplar<div class="im"><br>
characters for at least one language in the Unicode Common Locale<br>
Data Repository. [XXX What does this mean? -- Gerv]<br>
<br>
The Unicode CLDR project gathers information on the characters used in<br>
given languages, both the main characters, and those commonly used<br>
'foreign' characters.<br>
</div></blockquote>
<br>
Let me put my query another way: "what does this check add that is not covered by the previous checks"? Is it a way of expanding the definition of what's in a particular script, to include characters which are technically classed as being in other scripts? Or something else?<br>
</blockquote><div><br></div>No, it is to make the tests more specific to given languages, as a way of excluding unfamiliar characters from the same script. There are, for example, many Latin characters, more than most people realize; even excluding compatibility variants, it is over 1000. Someone might not realize that 'ꜱ' is not a regular '<font face="'times new roman', serif">s</font>', but a special small cap version, for example. There are a couple of ways to approach this problem.</div>
<div class="gmail_quote"><br></div><div class="gmail_quote">#39 provides categorizations of identifiers, see:</div><div class="gmail_quote"><a href="http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%3Alatn%3A%5D-%5B%3Anfkdqc%3Dn%3A%5D&g=identifier-restriction">http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%3Alatn%3A%5D-%5B%3Anfkdqc%3Dn%3A%5D&g=identifier-restriction</a></div>
<div class="gmail_quote"><br></div><div class="gmail_quote">CLDR provides information on which characters are used in which languages, allowing someone to limit characters to those supported by, for example, official languages or those supported in the UI of a product. (This may not be a good strategy IMO, but is a technique suitable for some environments.)</div>
<div class="gmail_quote"><br></div><div class="gmail_quote"><div><a href="http://unicode.org/repos/cldr-tmp/trunk/diff/by_type/misc.exemplarCharacters.html">http://unicode.org/repos/cldr-tmp/trunk/diff/by_type/misc.exemplarCharacters.html</a></div>
<div><br></div><div>(Both of these are generated from machine-readable data.)</div><div><br></div><div>Hope that helps,</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Gerv<br>
</blockquote></div><br>