1st Semester 2010/11: Assessing the Reliability of an Annotation Scheme for Indefinites

Instructors

Maria Aloni and Raquel Fernández
If you are interested in this project, please contact Maria by email.

ECTS

6

Description

Work in (computational) linguistics and cognitive science often builds on subjective judgements. Traditionally, it was considered sufficient to rely on the researchers' own interpretation of a few examples. In recent years, however, it has become widely accepted that further evidence is required that other individuals besides the researchers can reliably make the judgements underlying the work. Such evidence is typically gathered by measuring the extent to which several annotators agree on their judgements. In this project, we will study methods for assessing the reliability of linguistic judgements. In particular, we will investigate possible ways of evaluating the reliability of a functional map for indefinite expressions (such as any and some) characterising the possible functions that indefinites can fulfill. The project will consist in (1) designing an evaluation experiment for such a map of functions, (2) annotating some data according to the proposed design, and (3) evaluating the results by calculating inter-annotator agreement and by drawing conclusions on the reliability of the functional map under investigation.

Organisation

Tentative schedule:

• 1st week:
  • Introduction to the linguistic phenomenon: range of functions that indefinite expressions can fulfill and possible hypotheses as to their synchronic and diachronic distribution across different languages.
  • Design of the annotation experiment: formulation of guidelines and decision trees for the elicitation of judgements; selection of the data to be annotated.
• 2nd week:
  • Design of a web interface for carrying out the annotation; short pilot experiment.
  • Annotation of one single data set by several independent annotators.
• 3rd week:
  • Introduction to different methods for measuring inter-annotator agreement; students with some programming knowledge will be encouraged to write scripts to measure agreement automatically.
  • Calculation of inter-annotator agreement.
• 4th week:
  • Evaluation of the results obtained and writing up of a final (joint) paper.