Benchmarking Natural Language Understanding: A Psychological and Philosophical Perspective?

📝 Benchmarking Natural Language Understanding: A Psychological and Philosophical Perspective?

概要：Machine reading comprehension (MRC) has received considerable attention as a benchmark for natural language understanding. However, the conventional task design of MRC lacks explainability beyond the model interpretation, i.e., reading comprehension by a model cannot be explained in human terms. To this end, this talk provides a theoretical basis for the design of MRC datasets based on psychology as well as psychometrics, and summarizes it in terms of the prerequisites for benchmarking MRC. The talk may also include our recent (a little bit philosophical) discussion on language understanding and its evaluation.
[動画] [スライド] [論文1] (EACL 2021) [論文2] (ACL 2021)

※トークは日本語です。