<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:D="DAV:" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.gmailquote
{mso-style-name:gmail_quote;}
span.q
{mso-style-name:q;}
span.e
{mso-style-name:e;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:395976153;
mso-list-template-ids:-1571944742;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1
{mso-list-id:1019284269;
mso-list-template-ids:-492401654;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>I get – and agree with – what you’re saying
about changing things like bat from “Baltic (Other)” to “Baltic
Languages”. But to change mis from its current intensional meaning to “any
language” is not just broadening – it’s to change it into
something completely different altogether. It could appropriately be named “Other
Languages” – and getting rid of “other” changes it
utterly. It is one case where getting rid of the “other” aspect is
not appropriate because that is *<b>the* </b>defining criterion of the concept
it represents. <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>I would have little difficulty with the idea that we tell people
not to use mis. (Naturally -- again, at some point I suggested we not even
allow collections of any kind in IETF tags.) We could deprecate it in the
registry, or put a clear SHOULD NOT (if we think just maybe there are some
application scenarios in which informed and consenting adults might have a
reasonable use for it).<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>I’d have no problem with a request to the ISO 639 JAC for
a special-case ID for “any language” or “some unspecified
language”. (We’d need to clarify for them the difference between
this and und – which may or may not actually be a human language.)<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>For programming languages, I think a zxx variant would be a
reasonable solution.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Peter<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>
mark.edward.davis@gmail.com [mailto:mark.edward.davis@gmail.com] <b>On Behalf
Of </b>Mark Davis<br>
<b>Sent:</b> Monday, April 16, 2007 6:52 PM<br>
<b>To:</b> Peter Constable<br>
<b>Cc:</b> ietf-languages@iana.org; ltru@lists.ietf.org<br>
<b>Subject:</b> Re: [Ltru] Re: "mis" update review request<o:p></o:p></span></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><o:p> </o:p></p>
<div>
<p class=MsoNormal><span class=gmailquote>On 4/16/07, <b>Peter Constable</b>
<<a href="mailto:petercon@microsoft.com">petercon@microsoft.com</a>>
wrote:</span><o:p></o:p></p>
<div>
<div>
<p><span style='font-size:11.0pt;color:#1F497D'>Re 1: Yes, be careful: (a) the
majority of existing legacy usage of mis is bound to be in MARC, and (b) any
existing usage would assume the context of ISO 639-2 (i.e. mis in existing
usage is the exception list for ISO 639-2).</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:#1F497D'> </span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:#1F497D'>Re 2: The mis collection is
inherently unstable – unavoidably so. Prior to 2005-08-16, an
implementation of ISO 639-2 would have tagged Ainu content as mis; after that
date, an implementation of ISO 639-2 would have tagged Ainu content as ain;
existing content tagged before that date would not get retrieved by request for
ain, and it would be conformant to suppose that requests for mis would not
return Ainu content. The mis collection is ugly, pure and simple. So, I don't
see what the point is of getting worried over whether we're making mis
unstable: it's been that way for some time.</span><o:p></o:p></p>
</div>
</div>
<div>
<p class=MsoNormal><br>
What I'm saying is that <o:p></o:p></p>
<ol start=1 type=1>
<li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l1 level1 lfo1'>Right now in ISO 639-2, we have a number of
collections defined by exclusion, where XXX (Other) means any XXX that is
not already defined. Thus "bat" means "Any Baltic language
that doesn't already have a code". <o:p></o:p></li>
<li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l1 level1 lfo1'>Those collections are inherently unstable in ISO
639-2, since they contract each time a new XXX language is added.<o:p></o:p></li>
<li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l1 level1 lfo1'>The way to make an collection code XXX not
unstable is to make it not be not defined as an exclusion: removing the
(Other). [your proposal] <o:p></o:p></li>
<li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l1 level1 lfo1'>Then XXX is stable into the future, since adding
a new language of the type XXX doesn't affect it.<o:p></o:p></li>
<li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l1 level1 lfo1'>Thus if we change "bat" from Baltic
(Other)" into "Baltic", meaning any of the Baltic
languages, it becomes stable. <o:p></o:p></li>
<li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l1 level1 lfo1'>Such a change, being a broadening, can be carried
into BCP 47.<o:p></o:p></li>
<li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l1 level1 lfo1'>We can apply the same methodology to
"mis". That would change it from the fairly useless -- and
unstable -- "Any Language not otherwise encoded", into "Any
language". <o:p></o:p></li>
<li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l1 level1 lfo1'>It then becomes stable, and useful.<o:p></o:p></li>
</ol>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;
margin-left:4.8pt;margin-right:0in'>
<div>
<div>
<p><span style='font-size:11.0pt;color:#1F497D'>(Note: mis is badly defined
from a stability perspective, though I don't think there's much question of how
it's defined.)</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class=MsoNormal><br>
I agree that that is not the current definition of "mis" (see below).
<o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;
margin-left:4.8pt;margin-right:0in'>
<p class=MsoNormal><o:p> </o:p></p>
<div>
<div>
<p><span style='font-size:11.0pt;color:#1F497D'>Re 3(b): "</span>There are
times when detection can only determine that it looks like there is some
linguistic content -- it is not just binary data -- but current detection can't
really determine what it might be. That is, a code that means "according
to our best available detection methods this doesn't look like it is zxx".<span
style='font-size:11.0pt;color:#1F497D'>" If you want to use mis for that,
I would argue that that is significantly changing the semantics of mis. (Even
though mis is unstable, it is unstable on a qualitative level; this is a
categorical change.) I definitely oppose that. If you want an ID for
"undetermined human language", then that should be proposed. We
should not usurp an existing ID for that purpose.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class=MsoNormal><br>
It is a significant broadening of the semantics. And I'm not fixed on that. It
just seems that doing that broadening is congruent with the removal of
"(Other)" that you've proposed in other cases, and transforms a
useless and dangerous (for stability) code into a useful code. And since it is
a broadening, it is consistent with BCP 47. <br>
<br>
However, if that is too big a step to stomach, the alternative is to strongly
recommend that people never use "mis", and propose a new code for ISO
639-2 that has the meaning of "Any language". <o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;
margin-left:4.8pt;margin-right:0in'>
<p class=MsoNormal><o:p> </o:p></p>
<div>
<div>
<p><span style='font-size:11.0pt;color:#1F497D'>Re 4: I don't see how your
example differs from this: "Nous avons une phrase en français (but this is
in English)". The fact that the parenthetical text is in English doesn't
change the fact that the other text is in French. Similarly, in your example,
the fact that there is a comment in English does not change the fact that the
rest of the text is not in a human language. Do we create tags for "French
with embedded bits of English"?</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class=MsoNormal><br>
You have a good point. Again, I'm not hard and fast about this issue, but I
think there is definitely a significant distinction in usage between "this
is a chunk of stuff that looks like random binary data, like a JPEG", and
"this is stuff that looks like it might be written a programming language.",
a distinction that I think would be useful to provide for in BCP 47. On the
detection front, it is much easier to determine "this is random
binary", while not necessarily very easy to determine "this is a
programming language fragment". <o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;
margin-left:4.8pt;margin-right:0in'>
<p class=MsoNormal><o:p> </o:p></p>
<div>
<div>
<p><span style='font-size:11.0pt;color:#1F497D'>Peter</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:#1F497D'> </span><o:p></o:p></p>
<div style='border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0in 0in 0in;
border-color:-moz-use-text-color -moz-use-text-color'>
<p><b><span style='font-size:10.0pt'>From:</span></b><span style='font-size:
10.0pt'> <a href="mailto:mark.edward.davis@gmail.com" target="_blank">mark.edward.davis@gmail.com</a>
[mailto:<a href="mailto:mark.edward.davis@gmail.com" target="_blank">
mark.edward.davis@gmail.com</a>] <b>On Behalf Of </b>Mark Davis<br>
<b>Sent:</b> Monday, April 16, 2007 3:49 PM<br>
<b>To:</b> Peter Constable<br>
<b>Cc:</b> <a href="mailto:ietf-languages@iana.org" target="_blank">ietf-languages@iana.org</a>;
<a href="mailto:ltru@lists.ietf.org" target="_blank">ltru@lists.ietf.org</a><br>
<span class=q><b>Subject:</b> Re: [Ltru] Re: "mis" update review
request</span></span><o:p></o:p></p>
</div>
<p> <o:p></o:p></p>
<div>
<p>1. I think we have to be very careful here. The meaning of a standard like
ISO 639-2 is established not by <i>what we wish it would have said, </i>nor by <i>what
we would find out if we were able to read Peter's mind.</i> It is established
by the wording in the standard, and how reasonable people could interpret it.
The fact that "mis" was incorporated in order to account for MARC
codes is interesting, but is not in the text of the standard. We can't expect
users of BCP 47 to all be able to read Peter's mind before tagging. <br>
<br>
2. When we are looking at stability, that is very important: our goal is that
once content is correctly tagged, people can depend on the fact that we will
not change the meaning of a tag out from under them. So clarifications that we
add in future versions of 4646 or the registry are fine, as long as they do not
narrow the range of reasonable interpretations. We can broaden them. So in the
case of "mis", a proposed narrowing to include just the MARC codes is
clearly disallowed, since it was nowhere stated in ISO 639-2 at the time that
"mis" was added to the language registry (the BCP 47 semantics are
established at the time we add the code). That is one of the key principles of
BCP 47, is to isolate us where necessary from instabilities in the source
standards. <br>
<br>
(The one exception we might be able to make is where something is so badly
defined that most reasonable people couldn't come up with any consistent
definition for it.)<br>
<br>
3. Now, I think there are steps that can be taken to make the above moot. I
think Peter's suggestion for ISO 639-X of broadening all of the Collections to
remove the (Other) is exactly the right strategy, and if this can be done
before 4646bis is issued, all the better. So having <o:p></o:p></p>
<ul type=disc>
<li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l0 level1 lfo2'>aus Australian languages means
any of the languages on <a
href="http://www.ethnologue.com/show_family.asp?subid=90498"
target="_blank">http://www.ethnologue.com/show_family.asp?subid=90498</a> <o:p></o:p></li>
<li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l0 level1 lfo2'>bat Baltic (Other) => Baltic
languages, means any of the languages on <a
href="http://www.ethnologue.com/show_family.asp?subid=90207"
target="_blank">http://www.ethnologue.com/show_family.asp?subid=90207 </a><o:p></o:p></li>
<li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;
mso-list:l0 level1 lfo2'>mis Miscellaneous languages,
essentially the root for <a
href="http://www.ethnologue.com/family_index.asp" target="_blank">http://www.ethnologue.com/family_index.asp</a><o:p></o:p></li>
</ul>
<p style='margin-bottom:12.0pt'>and so on. This is useful on a number of
levels; it resolves a number of problems in the interpretation of language
codes, and makes the source standards themselves more stable. (In the ideal
case, we would have codes for each of the possible "decision points"
in the language tree. That is, if we look at any language code such as <a
href="http://www.ethnologue.com/show_lang_family.asp?code=eng" target="_blank">http://www.ethnologue.com/show_lang_family.asp?code=eng
</a>we'd have codes for each of the parent groupings, not just some of them,
like "Australian languages".) <br>
<br>
3. Randy raised the issue as to whether "mis" in the broad sense is
useful (as something that has linguistic content, but I don't know what it is).
It very much follows the model in #3. There are times when detection can only
determine that it looks like there is some linguistic content -- it is not just
binary data -- but current detection can't really determine what it might be.
That is, a code that means "according to our best available detection methods
this doesn't look like it is zxx". <br>
<br>
4. I'm leery of using zxx for programming languages, instead of just binary.
There is clearly some linguistic content in "if (content == null) { /*
remove the item in the lookup table */ ...}". Maybe we need another code
for this, something different than either 'art' or 'zxx'. <br>
<br>
Mark<o:p></o:p></p>
<div>
<p>On 4/14/07, <b>Peter Constable</b> <<a
href="mailto:petercon@microsoft.com" target="_blank">petercon@microsoft.com</a>>
wrote:<o:p></o:p></p>
<p>From: Randy Presuhn [mailto:<a href="mailto:randy_presuhn@mindspring.com"
target="_blank">randy_presuhn@mindspring.com</a>]<br>
<br>
<br>
> I find it very hard to believe that a reasonable analysis<br>
> (whether done by human or machine) would classify a text a <br>
> being "mis" without being able to recognize which of the<br>
> languages in that grouping the text belonged to. I can<br>
> believe someone could look at text and say "it's a slavic<br>
> language, but I'm not sure which one." Do we really think <br>
> someone or something would look at some text and say "it's<br>
> Ainu, Andamanese, or Etruscan, but I can't tell which, so<br>
> I'll tag it 'mis'"?<br>
<br>
If someone were so tempted, I would argue that would be inappropriate use of
mis. Since they do not know what it is, their declaration is that the language
identity is not determined, and the appropriate tag for that is und.
Appropriate use of mis does not require that one know the language of the
content; it does, however, require that one know it is *not* a language covered
by any of the available tags. <br>
<br>
<br>
<br>
Peter<br>
<br>
_______________________________________________<br>
Ltru mailing list<br>
<a href="mailto:Ltru@ietf.org" target="_blank">Ltru@ietf.org</a><br>
<a href="https://www1.ietf.org/mailman/listinfo/ltru" target="_blank">https://www1.ietf.org/mailman/listinfo/ltru
</a><o:p></o:p></p>
</div>
<p><br>
<br clear=all>
<br>
-- <br>
Mark <o:p></o:p></p>
</div>
</div>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
_______________________________________________<br>
Ltru mailing list<br>
<a href="mailto:Ltru@ietf.org">Ltru@ietf.org</a><br>
<a href="https://www1.ietf.org/mailman/listinfo/ltru" target="_blank">https://www1.ietf.org/mailman/listinfo/ltru</a><o:p></o:p></p>
</blockquote>
</div>
<p class=MsoNormal><br>
<br clear=all>
<br>
-- <br>
Mark <o:p></o:p></p>
</div>
</body>
</html>