A blog about the implications of the statistical properties of language
Observed agreement, expected agreement, and chins
Imaginez qu’au lieu de bien zyeuter les crânes…
Source: Yadav et al. (2017), Semantic relations in compound nouns: Perspectives from inter-annotator agreement
Question : Coucou, C’est quoi Cohen’s kappa ?
Réponse : Ce n’est pas moi, ce Cohen. 🙂 Cohen’s kappa est une façon de calculer l’accord entre deux personnes. Par exemple : Diap. 11, les deux personnes sont d’accord l’une avec l’autre 4/5, donc 80% des fois. Cohen’s kappa essaie de prendre en compte la possibilité d’être d’accord juste par hasard. On peut dire que la probabilité d’être d’accord juste par hasard, c’est 50% en ce cas. Imaginez qu’au lieu de bien zyeuter les crânes, chacun fait pile/face. On a donc ces possibilités :
Kevin oui, Mayla oui : 0.25
Kevin oui, Mayla non : 0.25
Kevin non, Mayla oui : 0.25
Kevin non, Mayla non : 0.25
Voilà les deux cas où ils sont d’accord :
Kevin oui, Mayla oui : 0.25
Kevin non, Mayla non : 0.25
0.25 + 0.25, cela fait 0.50. Donc, ils peuvent être d’accord 0.50 (50%) des fois… par hasard–rappelez-vous qu’ils ont fait pile/face.
Le calcul du kappa de Cohen, cela se fait comme ça :
…donc on prévoit que le kappa de Cohen, c’est toujours moins que l’accord “cru.” Vu que l’on ne veut pas surestimer la performance, on utilise le kappa de Cohen pour éviter cela.
Question : C’est un genre de coefficient correcteur de réalité j’ai l’impression pour ne pas s’emballer sur des résultats in vivo qui parfois peuvent être faussés par le caractère très humains des annotateurs. Enfin je crois comprendre ça 🙂
Réponse : Tu l’as dit mieux que moi ! 🙂
Why would anyone want to label skull specimens as to whether or not they have chins? See this post, and be sure to read MELewis‘s comment.
French notes
l’accord inter-annotateur : inter-annotator agreement. See a nice set of slides on the topic, in French, here.
zyeuter : to scrutinize. It seems like a soutenu word, but given the book where I ran across it, I would expect it to be familier. (Comment from native speaker Phil dAnge: zyeuter is definitively not “soutenu”but completely “familier, limite argot ”)
15 thoughts on “Observed agreement, expected agreement, and chins”
Bravo, belle démonstration de maîtrise du français . Your French is better than my English . Two notes : the choice of “cru”(l’accord cru) is not the best in this situation . Better to write “l’accord attendu”, as we don’t have a single word for expected, or “l’accord prévu” if the person was nearly sure of the result by advance .
And zyeuter is definitively not “soutenu”but completely “familier, limite argot ” .
There are many . Soutenu : “regarder” of course, then “examiner”and “scruter”, these two imply more attention, “fixer”,this one implies no other movement, “mirer”, this one is older, “contempler”, like gaze at. Familier : “mater” (very common), “reluquer”, “lorgner”, “bigler”,”guigner” (older but still used), “viser” (yes colloquially it means simply regarder). Now there are also the multiple classical slangs of different social backgrounds – and there are several – plus nowadays the “slangs”from the brainless teens influenced by American money makers or immigration suburbs . I don’t know their slangs, each time I see one I shoot before he can open his annoying mouth .
Note : pardon me I answered your question about “raw agreement” in a wrong place. I hope you can get it nevertheless . Oh, sometimes a “war agreement” can be a “raw agreement” with another meaning isn’t it ?
Yes, pardon me I speak with you as I’d do with a French mate . You’re right, it means “borderline slang, it is a rather modern colloquial way of speaking . You’ll hear it often nowadays : ” Le mec était vraiment en colère, limite agressif tu vois …”, this sort of things .
Thanks! What I was looking for is what you would call in English “raw agreement,” i.e. the agreement before it’s been adjusted—in this formula, the “observed agreement,” calculated as 4/5 in the example. What’s the word for “raw” in this context?
I had to think . There are several possibilities, of course depending on more subtle notions in the speaker’s mind . For seen before adjustment there is a very decent term, “estimé” estimated but this one implies a minimum of thinking . For an immediate instinctive estimation there is the very common phrase ” à première vue”, with its very “familier” equivalent “à la louche”.
In your sentence I would say “…que l’accord apparent OR l’accord supposé à première vue”.
Of course in England an insult to an Upper Class Twit is to call them a ‘chinless wonder’. And Upper Class Twit is one of high birth who appears to have no discernable grasp of reality.
We have altogether too much ‘chin’ where I come from, although no upper class roots. Agreement could never be predicted as we are far too liable to disagree for the fun of it!
Zipf's Law 
Bravo, belle démonstration de maîtrise du français . Your French is better than my English . Two notes : the choice of “cru”(l’accord cru) is not the best in this situation . Better to write “l’accord attendu”, as we don’t have a single word for expected, or “l’accord prévu” if the person was nearly sure of the result by advance .
And zyeuter is definitively not “soutenu”but completely “familier, limite argot ” .
LikeLiked by 1 person
…and I updated the “French notes” with your comment about zyeuter.
LikeLiked by 1 person
Is there a more high-register alternative to zyeuter?
LikeLike
There are many . Soutenu : “regarder” of course, then “examiner”and “scruter”, these two imply more attention, “fixer”,this one implies no other movement, “mirer”, this one is older, “contempler”, like gaze at. Familier : “mater” (very common), “reluquer”, “lorgner”, “bigler”,”guigner” (older but still used), “viser” (yes colloquially it means simply regarder). Now there are also the multiple classical slangs of different social backgrounds – and there are several – plus nowadays the “slangs”from the brainless teens influenced by American money makers or immigration suburbs . I don’t know their slangs, each time I see one I shoot before he can open his annoying mouth .
Note : pardon me I answered your question about “raw agreement” in a wrong place. I hope you can get it nevertheless . Oh, sometimes a “war agreement” can be a “raw agreement” with another meaning isn’t it ?
LikeLiked by 1 person
To clarify: “limite argot,” that means “borderline slang”? (Never heard “limite” before…)
LikeLike
Yes, pardon me I speak with you as I’d do with a French mate . You’re right, it means “borderline slang, it is a rather modern colloquial way of speaking . You’ll hear it often nowadays : ” Le mec était vraiment en colère, limite agressif tu vois …”, this sort of things .
LikeLiked by 2 people
And I love it. 🙂
LikeLiked by 1 person
Thanks! What I was looking for is what you would call in English “raw agreement,” i.e. the agreement before it’s been adjusted—in this formula, the “observed agreement,” calculated as 4/5 in the example. What’s the word for “raw” in this context?
LikeLike
I had to think . There are several possibilities, of course depending on more subtle notions in the speaker’s mind . For seen before adjustment there is a very decent term, “estimé” estimated but this one implies a minimum of thinking . For an immediate instinctive estimation there is the very common phrase ” à première vue”, with its very “familier” equivalent “à la louche”.
In your sentence I would say “…que l’accord apparent OR l’accord supposé à première vue”.
LikeLiked by 2 people
Of course in England an insult to an Upper Class Twit is to call them a ‘chinless wonder’. And Upper Class Twit is one of high birth who appears to have no discernable grasp of reality.
LikeLiked by 3 people
Thank you for the lesson of poetic colloquial English .
LikeLiked by 2 people
You have no IDEA how many times I’ve muttered it about someone or other 😉
LikeLiked by 3 people
Trying to figure out whether or not one could use “chinless wonder” to refer to Trump–he’s not at all classy, though…
LikeLiked by 1 person
We have altogether too much ‘chin’ where I come from, although no upper class roots. Agreement could never be predicted as we are far too liable to disagree for the fun of it!
LikeLike