Fiche du document
5 juin 2019
- doi: 10.4000/books.aaccademia.4487
- isbn: 9788831978422
- eisbn: 9788831978699
https://creativecommons.org/licenses/by-nc-nd/4.0/ , info:eu-repo/semantics/openAccess
Mots-clés
elaborazione del linguaggio naturale strumenti del linguaggio traitement du langage naturel outils du langage Natural Language Processing Speech Tools Language & Linguistics LAN009000 CFSujets proches
Assessment DialogCiter ce document
Francesco Cutugno et al., « Overview of the EVALITA 2018 Evaluation of Italian DIALogue systems (IDIAL) Task », Accademia University Press, ID : 10.4000/books.aaccademia.4487
Métriques
Partage / Export
Résumé
We report about the organization of the IDIAL (Evaluation of Italian DIALogue systems) task at EVALITA 2018, the first shared task aiming at assessing interactive characteristics of conversational agents for the Italian language. In this perspective, IDIAL considers a dialogue system as a “black box” (i.e., evaluation can not access internal components of the system), and measures the system performance on three dimensions: task completion, effectiveness of the dialogue and user satisfaction. We describe the IDIAL evaluation protocol, and show how it has been applied to the three participating systems. Finally, we briefly discuss current limitations and future improvements of the IDIAL methodology.