Does It Make Sense? And Why? A Pilot Study For Sense Making And Explanation

Cunxiang Wang,Shuailong Liang,Yue Zhang,Xiaonan Li,Tian Gao
DOI: https://doi.org/10.18653/v1/p19-1393
2019-01-01
Abstract:Introducing common sense to natural language understanding systems has received increasing research attention. It remains a fundamental question on how to evaluate whether a system has a sense making capability. Existing benchmarks measures commonsense knowledge indirectly and without explanation. In this paper, we release a benchmark to directly test whether a system can differentiate natural language statements that make sense from those that do not make sense. In addition, a system is asked to identify the most crucial reason why a statement does not make sense. We evaluate models trained over large-scale language modeling tasks as well as human performance, showing that there are different challenges for system sense making.
What problem does this paper attempt to address?