************************************************************************
IMPLEMENTATION ASSOCIATED WITH COLING-2014 PAPER
"Empirical Analysis of Aggregation Methods for Collective Annotation"
by Ciyang Qing, Ulle Endriss, Raquel Fernandez and Justin Kruger
************************************************************************

This directory contains an implementation (in R) of the aggregation 
rules used in the paper. It consists if the following source files:

- main_coling.R:

  The main script that produces the main results for the 3 datasets in 
  the paper.

  In an R console, type the following command:
  
      source("main_coling.R")    

- aux.R:
  
  Auxiliary functions that facilitate the definitions of the aggregators.

- aggregators.R:

  Implementations of the aggregators proposed in the paper, together with 
  some variations.

- evaluation.R: 

  Functions that are used in evaluating the aggregators against the gold 
  standard.

- demo.Rmd 

  R Markdown file that generates the demo file demo.html. 

- demo.html:
 
  Demo of the aggregators.

************************************************************************