Observed agreement, expected agreement, and chins

Imaginez qu’au lieu de bien zyeuter les crânes…

Screenshot 2017-08-29 06.26.00
Source: Yadav et al. (2017), Semantic relations in compound nouns: Perspectives from inter-annotator agreement

Question : Coucou, C’est quoi Cohen’s kappa ?

Réponse : Ce n’est pas moi, ce Cohen.  🙂  Cohen’s kappa est une façon de calculer l’accord entre deux personnes.  Par exemple : Diap. 11, les deux personnes sont d’accord l’une avec l’autre 4/5, donc 80% des fois.  Cohen’s kappa essaie de prendre en compte la possibilité d’être d’accord juste par hasard.  On peut dire que la probabilité d’être d’accord juste par hasard, c’est 50% en ce cas.  Imaginez qu’au lieu de bien zyeuter les crânes, chacun fait pile/face.  On a donc ces possibilités :

  • Kevin oui, Mayla oui : 0.25
  • Kevin oui, Mayla non : 0.25
  • Kevin non, Mayla oui : 0.25
  • Kevin non, Mayla non : 0.25
Voilà les deux cas où ils sont d’accord :
  • Kevin oui, Mayla oui : 0.25
  • Kevin non, Mayla non : 0.25
0.25 + 0.25, cela fait 0.50.  Donc, ils peuvent être d’accord 0.50 (50%) des fois… par hasard–rappelez-vous qu’ils ont fait pile/face.
Le calcul du kappa de Cohen, cela se fait comme ça :
  • Accord experimenté – Accord prévu par hasard = 0.80 – 0.50 = 0.3
…divisé par accord prévu par hasard, donc
  • 0.3 / 0.5 = 0.6
…donc on prévoit que le kappa de Cohen, c’est toujours moins que l’accord “cru.”  Vu que l’on ne veut pas surestimer la performance, on utilise le kappa de Cohen pour éviter cela.

Question :  C’est un genre de coefficient correcteur de réalité j’ai l’impression pour ne pas s’emballer sur des résultats in vivo qui parfois peuvent être faussés par le caractère très humains des annotateurs.  Enfin je crois comprendre ça 🙂

Réponse : Tu l’as dit mieux que moi !  🙂
Why would anyone want to label skull specimens as to whether or not they have chins?  See this post, and be sure to read MELewis‘s comment.


French notes

l’accord inter-annotateur : inter-annotator agreement.  See a nice set of slides on the topic, in French, here.
zyeuter : to scrutinize.  It seems like a soutenu word, but given the book where I ran across it, I would expect it to be familier.  (Comment from native speaker Phil dAnge: zyeuter is definitively not “soutenu”but completely “familier, limite argot ”)

15 thoughts on “Observed agreement, expected agreement, and chins”

  1. Bravo, belle démonstration de maîtrise du français . Your French is better than my English . Two notes : the choice of “cru”(l’accord cru) is not the best in this situation . Better to write “l’accord attendu”, as we don’t have a single word for expected, or “l’accord prévu” if the person was nearly sure of the result by advance .
    And zyeuter is definitively not “soutenu”but completely “familier, limite argot ” .

    Liked by 1 person

      1. There are many . Soutenu : “regarder” of course, then “examiner”and “scruter”, these two imply more attention, “fixer”,this one implies no other movement, “mirer”, this one is older, “contempler”, like gaze at. Familier : “mater” (very common), “reluquer”, “lorgner”, “bigler”,”guigner” (older but still used), “viser” (yes colloquially it means simply regarder). Now there are also the multiple classical slangs of different social backgrounds – and there are several – plus nowadays the “slangs”from the brainless teens influenced by American money makers or immigration suburbs . I don’t know their slangs, each time I see one I shoot before he can open his annoying mouth .
        Note : pardon me I answered your question about “raw agreement” in a wrong place. I hope you can get it nevertheless . Oh, sometimes a “war agreement” can be a “raw agreement” with another meaning isn’t it ?

        Liked by 1 person

      1. Yes, pardon me I speak with you as I’d do with a French mate . You’re right, it means “borderline slang, it is a rather modern colloquial way of speaking . You’ll hear it often nowadays : ” Le mec était vraiment en colère, limite agressif tu vois …”, this sort of things .

        Liked by 2 people

  2. Thanks! What I was looking for is what you would call in English “raw agreement,” i.e. the agreement before it’s been adjusted—in this formula, the “observed agreement,” calculated as 4/5 in the example. What’s the word for “raw” in this context?


  3. I had to think . There are several possibilities, of course depending on more subtle notions in the speaker’s mind . For seen before adjustment there is a very decent term, “estimé” estimated but this one implies a minimum of thinking . For an immediate instinctive estimation there is the very common phrase ” à première vue”, with its very “familier” equivalent “à la louche”.
    In your sentence I would say “…que l’accord apparent OR l’accord supposé à première vue”.

    Liked by 2 people

  4. Of course in England an insult to an Upper Class Twit is to call them a ‘chinless wonder’. And Upper Class Twit is one of high birth who appears to have no discernable grasp of reality.

    Liked by 3 people

  5. We have altogether too much ‘chin’ where I come from, although no upper class roots. Agreement could never be predicted as we are far too liable to disagree for the fun of it!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Curative Power of Medical Data

JCDL 2020 Workshop on Biomedical Natural Language Processing


Criminal Curiosities


Biomedical natural language processing

Mostly Mammoths

but other things that fascinate me, too


Adventures in natural history collections

Our French Oasis


ACL 2017

PC Chairs Blog

Abby Mullen

A site about history and life

EFL Notes

Random commentary on teaching English as a foreign language

Natural Language Processing

Université Paris-Centrale, Spring 2017

Speak Out in Spanish!

living and loving language




Exploring and venting about quantitative issues

%d bloggers like this: