Subtask 1 will be evaluated as in the Le.Wi.Di shared tasks at SemEval 2021 and 2023 about learning with disagreement (Uma et al. 2021 and Leonardelli et al. 2023). First, the models that output hard labels will be compared to the gold standard by using the standard classification metric F1. The second evaluation metric will be the cross-entropy between the system soft label values and the soft labels generated from the annotators average votes.

Subtask 2 is a binary hierarchical classification problem. We will use the ICM metric (Amigó and Delgado, 2022), an information theoretic based metric which considers both the hierarchical structure and the class specificity. It is applicable to both hard and soft labels. The ICM metric will be the official metric considered for the ranking for both hard labels (ICM) and soft labels (ICM Soft).

The implementation of the official metrics (F1, cross-entropy and ICM) will be based on PyEvALL. An example of the evaluation can be found in Examples.ipynb.

For both subtasks 1 and 2, we will use non-informative baselines that classify all instances as either the majority or the minority classes (“BASELINE_all_ones” and “BASELINE_all_zeros”). Furthermore, we also use a random classifier “BASELINE_random_classifier”, a Term Frequency-Inverse Document Frequency (TFIDF) with Support Vector Classifier (SVC) “BASELINE_tfidf_svc”, Fast Text embedding with SVC “BASELINE_fast_text_svc”, and a fine-tuned BERT model (Cañete et al. 2020) “BASELINE_beto”. The baselines and code to produce them can be found at clic-ub/DETESTS-Dis

Results

Task 1 with Hard Labels

Task 1 with Soft Labels

Task 2 with Hard Labels

Task 2 with Soft Labels

References

Amigó, E., & Delgado, A. (2022, May). Evaluating extreme hierarchical multi-label classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5809-5819).

Cañete, J., Chaperon, G., Fuentes, R., Ho, J. H., Kang, H., & Pérez, J. (2020). Spanish pre-trained bert model and evaluation data. In PML4DC at ICLR 2020.

Leonardelli, E., Uma, A., Abercrombie, G., Almanea, D., Basile, V., Fornaciari, T., Plank, B., Rieser, V., Uma, A., & Poesio, M. (2023). SemEval-2023 Task 11: Learning With Disagreements (LeWiDi). In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2304–2318. Association for Computational Linguistics.

Uma, A., Fornaciari, T., Dumitrache, A., Miller, T., Chamberlain, J., Plank, B., Simpson, E. & Poesio, M. (2021). ‘SemEval-2021 Task 12: Learning with Disagreements’. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021) (pp. 338-347). Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/2021.semeval-1.41