[Ltru] Re: "mis" update review request

Tue Apr 17 03:51:50 CEST 2007

On 4/16/07, Peter Constable <petercon at microsoft.com> wrote:
>
>  Re 1: Yes, be careful: (a) the majority of existing legacy usage of mis
> is bound to be in MARC, and (b) any existing usage would assume the context
> of ISO 639-2 (i.e. mis in existing usage is the exception list for ISO
> 639-2).
>
>
>
> Re 2: The mis collection is inherently unstable – unavoidably so. Prior to
> 2005-08-16, an implementation of ISO 639-2 would have tagged Ainu content as
> mis; after that date, an implementation of ISO 639-2 would have tagged Ainu
> content as ain; existing content tagged before that date would not get
> retrieved by request for ain, and it would be conformant to suppose that
> requests for mis would not return Ainu content. The mis collection is ugly,
> pure and simple. So, I don't see what the point is of getting worried over
> whether we're making mis unstable: it's been that way for some time.
>

What I'm saying is that

   1. Right now in ISO 639-2, we have a number of collections defined by
   exclusion, where XXX (Other) means any XXX that is not already defined. Thus
   "bat" means "Any Baltic language that doesn't already have a code".
   2. Those collections are inherently unstable in ISO 639-2, since they
   contract each time a new XXX language is added.
   3. The way to make an collection code XXX not unstable is to make it
   not be not defined as an exclusion: removing the (Other). [your proposal]
   4. Then XXX is stable into the future, since adding a new language of
   the type XXX doesn't affect it.
   5. Thus if we change "bat" from Baltic (Other)" into "Baltic", meaning
   any of the Baltic languages, it becomes stable.
   6. Such a change, being a broadening, can be carried into BCP 47.
   7. We can apply the same methodology to "mis". That would change it
   from the fairly useless -- and unstable -- "Any Language not otherwise
   encoded", into "Any language".
   8. It then becomes stable, and useful.

(Note: mis is badly defined from a stability perspective, though I don't
> think there's much question of how it's defined.)
>

I agree that that is not the current definition of "mis" (see below).

Re 3(b): "There are times when detection can only determine that it looks
> like there is some linguistic content -- it is not just binary data -- but
> current detection can't really determine what it might be. That is, a code
> that means "according to our best available detection methods this doesn't
> look like it is zxx"." If you want to use mis for that, I would argue that
> that is significantly changing the semantics of mis. (Even though mis is
> unstable, it is unstable on a qualitative level; this is a categorical
> change.) I definitely oppose that. If you want an ID for "undetermined human
> language", then that should be proposed. We should not usurp an existing ID
> for that purpose.
>

It is a significant broadening of the semantics. And I'm not fixed on that.
It just seems that doing that broadening is congruent with the removal of
"(Other)" that you've proposed in other cases, and transforms a useless and
dangerous (for stability) code into a useful code. And since it is a
broadening, it is consistent with BCP 47.

However, if that is too big a step to stomach, the alternative is to
strongly recommend that people never use "mis", and propose a new code for
ISO 639-2 that has the meaning of "Any language".

Re 4: I don't see how your example differs from this: "Nous avons une phrase
> en français (but this is in English)". The fact that the parenthetical text
> is in English doesn't change the fact that the other text is in French.
> Similarly, in your example, the fact that there is a comment in English does
> not change the fact that the rest of the text is not in a human language. Do
> we create tags for "French with embedded bits of English"?
>

You have a good point. Again, I'm not hard and fast about this issue, but I
think there is definitely a significant distinction in usage between "this
is a chunk of stuff that looks like random binary data, like a JPEG", and
"this is stuff that looks like it might be written a programming language.",
a distinction that I think would be useful to provide for in BCP 47. On the
detection front, it is much easier to determine "this is random binary",
while not necessarily very easy to determine "this is a programming language
fragment".

Peter
>
>
>
> *From:* mark.edward.davis at gmail.com [mailto:mark.edward.davis at gmail.com] *On
> Behalf Of *Mark Davis
> *Sent:* Monday, April 16, 2007 3:49 PM
> *To:* Peter Constable
> *Cc:* ietf-languages at iana.org; ltru at lists.ietf.org
> *Subject:* Re: [Ltru] Re: "mis" update review request
>
>
>
> 1. I think we have to be very careful here. The meaning of a standard like
> ISO 639-2 is established not by *what we wish it would have said, *nor by
> *what we would find out if we were able to read Peter's mind.* It is
> established by the wording in the standard, and how reasonable people could
> interpret it. The fact that "mis" was incorporated in order to account for
> MARC codes is interesting, but is not in the text of the standard. We can't
> expect users of BCP 47 to all be able to read Peter's mind before tagging.
>
> 2. When we are looking at stability, that is very important: our goal is
> that once content is correctly tagged, people can depend on the fact that we
> will not change the meaning of a tag out from under them. So clarifications
> that we add in future versions of 4646 or the registry are fine, as long as
> they do not narrow the range of reasonable interpretations. We can broaden
> them. So in the case of "mis", a proposed narrowing to include just the MARC
> codes is clearly disallowed, since it was nowhere stated in ISO 639-2 at the
> time that "mis" was added to the language registry (the BCP 47 semantics are
> established at the time we add the code). That is one of the key principles
> of BCP 47, is to isolate us where necessary from instabilities in the source
> standards.
>
> (The one exception we might be able to make is where something is so badly
> defined that most reasonable people couldn't come up with any consistent
> definition for it.)
>
> 3. Now, I think there are steps that can be taken to make the above moot.
> I think Peter's suggestion for ISO 639-X of broadening all of the
> Collections to remove the (Other) is exactly the right strategy, and if this
> can be done before 4646bis is issued, all the better. So having
>
>    - aus    Australian languages means any of the languages on
>    http://www.ethnologue.com/show_family.asp?subid=90498
>    - bat    Baltic (Other) => Baltic languages, means any of the
>    languages on http://www.ethnologue.com/show_family.asp?subid=90207
>    - mis    Miscellaneous languages, essentially the root for
>    http://www.ethnologue.com/family_index.asp
>
> and so on. This is useful on a number of levels; it resolves a number of
> problems in the interpretation of language codes, and makes the source
> standards themselves more stable. (In the ideal case, we would have codes
> for each of the possible "decision points" in the language tree. That is, if
> we look at any language code such as
> http://www.ethnologue.com/show_lang_family.asp?code=eng we'd have codes
> for each of the parent groupings, not just some of them, like "Australian
> languages".)
>
> 3. Randy raised the issue as to whether "mis" in the broad sense is useful
> (as something that has linguistic content, but I don't know what it is). It
> very much follows the model in #3. There are times when detection can only
> determine that it looks like there is some linguistic content -- it is not
> just binary data -- but current detection can't really determine what it
> might be. That is, a code that means "according to our best available
> detection methods this doesn't look like it is zxx".
>
> 4. I'm leery of using zxx for programming languages, instead of just
> binary. There is clearly some linguistic content in "if (content == null) {
> /* remove the item in the lookup table */ ...}". Maybe we need another code
> for this, something different than either 'art' or 'zxx'.
>
> Mark
>
> On 4/14/07, *Peter Constable* <petercon at microsoft.com> wrote:
>
> From: Randy Presuhn [mailto:randy_presuhn at mindspring.com]
>
>
> > I find it very hard to believe that a reasonable analysis
> > (whether done by human or machine) would classify a text a
> > being "mis" without being able to recognize which of the
> > languages in that grouping the text belonged to.  I can
> > believe someone could look at text and say "it's a slavic
> > language, but I'm not sure which one."  Do we really think
> > someone or something would look at some text and say "it's
> > Ainu, Andamanese, or Etruscan, but I can't tell which, so
> > I'll tag it 'mis'"?
>
> If someone were so tempted, I would argue that would be inappropriate use
> of mis. Since they do not know what it is, their declaration is that the
> language identity is not determined, and the appropriate tag for that is
> und. Appropriate use of mis does not require that one know the language of
> the content; it does, however, require that one know it is *not* a language
> covered by any of the available tags.
>
>
>
> Peter
>
> _______________________________________________
> Ltru mailing list
> Ltru at ietf.org
> https://www1.ietf.org/mailman/listinfo/ltru
>
>
>
>
> --
> Mark
>
> _______________________________________________
> Ltru mailing list
> Ltru at ietf.org
> https://www1.ietf.org/mailman/listinfo/ltru
>
>

-- 
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.alvestrand.no/pipermail/ietf-languages/attachments/20070416/eb6413f2/attachment-0001.html