Modal Sense Classification At Large. Paraphrase-Driven Sense Projection, Semantically Enriched Classification Models and Cross-Genre Evaluations

Ana Marasović, Mengfei Zou, Alexis Palmer, Anette Frank


Modal verbs have different interpretations depending on their context. Their sense categories – epistemic, deontic and dynamic – provide important dimensions of meaning for the interpretation of discourse.

Previous work on modal sense classification achieved relatively high performance using shallow lexical and syntactic features drawn from small-size annotated corpora. Due to the restricted empirical basis, it is difficult to assess the particular difficulties of modal sense classification and the generalization capacity of the proposed models.

In this work we create large-scale, high-quality annotated corpora for modal sense classification using an automatic paraphrase-driven projec- tion approach. Using the acquired corpora, we investigate the modal sense classification task from different perspectives.

We uncover the difficulty of specific sense distinctions by investigat-ing distributional bias and reducing the sparsity of existing small-scale corpora used in prior work. We build a semantically enriched model for modal sense classification by designing novel features related to lexical, proposition-level and discourse-level semantic factors. Besides improved classification performance, closer examination of interpretable feature sets unveils relevant semantic and contextual factors in modal sense classification. Finally, we investigate genre effects on modal sense distribution and how they affect classification performance.

Our investigations uncover the difficulty of specific sense distinctions and how they are affected by training set size and distributional bias. Our large-scale experiments confirm that semantically enriched models outperform models built on shallow feature sets. Cross-genre experiments shed light on differences in sense distributions across genres and confirm that semantically enriched models have high generalization capacity, especially in unstable distributional settings. 


semantics; machine learning; annotation; modality;

Full Text:



Baayen, Harald R., Richard Piepenbrock, and Leon Gulikers. 1996. CELEX2. Philadelphia: Linguistic Data Consortium.

Baker, Kathryn, Michael Bloodgood, Bonnie J. Dorr, Chris Callison-Burch, Nathaniel W. Filardo, Christine Piatko, Lori Levin, and Scott Miller. 2012. Modality and Negation in SIMT Use of Modality and Negation in Semantically-Informed Syntactic MT. Computational Linguistics 38(2):411–438.

Baker, Kathryn, Michael Bloodgood, Bonnie J Dorr, Nathaniel W Filardo, Lori Levin, and Christine Piatko. 2010. A Modality Lexicon and its use in Automatic Tagging. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC-2010), pages 1402– 1407.

Becker, Maria, Alexis Palmer, and Anette Frank. 2016. Clause types and modality in argumentative microtexts. In Workshop on Foundations of the Language of Argumentation (in conjunction with COMMA 2016). Potsdam, Germany.

Benamara, Farah, Baptiste Chardon, Yannick Mathieu, Vladimir Popescu, and Nicholas Asher. 2012. How do negation and modality impact on opinions? In Proceedings of the Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics, pages 10–18. Jeju, Republic of Korea: Association for Computational Linguistics.

Bird, Steven, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python. O’Reilly Media, Inc.

Bloodgood, Michael and K Vijay-Shanker. 2009. Taking into account the differences between actively and passively acquired data: The case of active learning with support vector machines for imbalanced datasets. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers, pages 137–140.

Cohen, J. 1960. A coefficient for agreement for nominal scales. Education and Psychological Measurement 20(1):37–46.

Cui, Yanyan and Ting Chi. 2013. Annotating Modal Expressions in the Chinese Treebank. In Proceedings of IWCS 2013 Workshop on Annotation of Modal Meanings in Natural Language (WAMM), pages 24–32. Potsdam, Germany.

De Marneffe, Marie-Catherine and Christopher D Manning. 2008. The Stanford typed dependencies representation. In Coling 2008: Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation, pages 1–8. Association for Computational Linguistics.

de Marneffe, Marie-Catherine, Christopher D. Manning, and Christopher Potts. 2011. Veridicality and Utterance Understanding. In Proceedings of the Fifth International Conference on Semantic Computing at IEEE 2011, pages 430–437. IEEE.

de Marneffe, Marie-Catherine, Christopher D. Manning, and Christopher Potts. 2012. Did It Happen? The Pragmatic Complexity of Veridicality Assessment. Computational Linguistics 38(2):301–333.

Diab, Mona and Philip Resnik. 2002. An Unsupervised Method for Word Sense Tagging using Parallel Corpora. In Proceedings of 40th Annual Meet- ing of the Association for Computational Linguistics (ACL-2002), pages 255–262. Philadelphia, Pennsylvania, USA.

Fellbaum, Christiane. 1999. WordNet. Wiley Online Library.

Fleiss, J. L. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin 76(5):378–382.

Friedrich, Annemarie and Alexis Palmer. 2014. Automatic prediction of aspectual class of verbs in context. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL-2014).

Ganitkevitch, Juri, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The Paraphrase Database. In Proceedings of the 51st Annual Meet- ing of the Association for Computational Linguistics (ACL-HLT 2013), pages 758–764. Atlanta, Georgia.

Graca, Joao V., Kuzman Ganchev, and Ben Taskar. 2007. Expectation maximization and posterior constraints. In Advances in Neural Information Processing Systems (NIPS).

Hendrickx, Iris, Amália Mendes, and Silvia Mencarelli. 2012. Modality in Text: a Proposal for Corpus Annotation. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC- 2012), pages 1805–1812. Istanbul, Turkey.

Hwa, Rebecca, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Ko- lak. 2005. Bootstrapping Parsers via Syntactic Projection across Parallel Texts. Natural Language Engineering 11(3):311–325.

Ide, Nancy, Collin Baker, Christiane Fellbaum, and Charles Fillmore. 2008. MASC: The manually annotated sub-corpus of American English. In Pro- ceedings of the Sixth International Conference on Language Resources and Evaluation (LREC-2008).

Koehn, Philipp. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the 2005 Machine Translation Summit, pages 79–86.

Kratzer, Angelika. 1981. The Notional Category of Modality. In H. J. Eikmeyer and H. Rieser, eds., Words, worlds, and contexts: New approaches in word semantics, pages 38–74. Berlin: de Gruyter.

Kratzer, Angelika. 1991. Modality. In A. von Stechow and D. Wunderlic, eds., Semantics: An International Handbook of Contemporary Research, pages 639–650. Berlin: de Gruyter.

Krippendorff, Klaus. 2004. Measuring the reliability of qualitative text analysis data. In Annenberg School for Communication Department Papers. University of Pennsylvania.

References / 51

Kullback, Solomon and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics 22(1):79–86.

Lee, Kenton, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event detection and factuality assessment with non-expert supervision. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP-2015), pages 1643–1648. Lisbon, Portugal.

Light, Mark, Xin Y. Qiu, and Padmini Srinivasan. 2004. The language of bioscience: Facts, speculations, and statements in between. In Proceedings of BioLINK 2004.

Loaiciga, Sharid, Thomas Meyer, and Andrei Popescu-Belis. 2014. English- French Verb Phrase Alignment in Europarl. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC- 2014).

Manning, Christopher D, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60.

Marasović, Ana and Anette Frank. 2016. Multilingual Modal Sense Classification using a Convolutional Neural Network. In First Workshop on Representation Learning for NLP. Berlin, Germany.

McNemar, Quinn. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2):153–157.

Morante, Roser and Walter Daelemans. 2011. Annotating modality and negation for a machine reading evaluation. In CLEF 2011 Labs and Workshop Notebook Papers.

Morante, Roser and Caroline Sporleder. 2012. Modality and Negation: An Introduction to the Special Issue. Computational Linguistics 38(2):223– 260.

Nissim, Malvina, Paola Pietrandrea, Andrea Sanso, and Caterina Mauri. 2013. Cross-linguistic annotation of modality: a data-driven hierarchical model. In Proceedings of the 9th Joint ISO - ACL SIGSEM Workshop on Interoperable Semantic Annotation, pages 7–14. Potsdam, Germany.

Padó, Sebastian and Mirella Lapata. 2009. Cross-lingual annotation projection of semantic roles. Journal of Artificial Intelligence Research 36(1):307–340.

Passonneau, Rebecca J., Nancy Ide, Songqiao Su, and Jesse Stuart. 2014. Biber Redux: Reconsidering Dimensions of Variation in American English. In Proceedings of the 25th International Conference on Computational Lin- guistics (COLING-2014).

Prabhakaran, Vinodkumar, Michael Bloodgood, Mona Diab, Bonnie Dorr, Lori Levin, Christine D. Piatko, Owen Rambow, and Benjamin Van Durme. 2012. Statistical modality tagging from rule-based annotations and crowdsourcing. In Proceedings of the Workshop on Extra- Propositional Aspects of Meaning in Computational Linguistics, pages 57– 64. Jeju, Republic of Korea.

Reiter, Nils and Anette Frank. 2010. Identifying Generic Noun Phrases. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL-2010), pages 40–49. Uppsala, Sweden.

Resnik, Philip. 2004. Exploiting hidden meanings: Using bilingual text for monolingual annotation. In A. Gelbukh, ed., Computational Linguistics and Intelligent Text Processing, no. 2945 in Lecture Notes in Computer Science, pages 283–299. Springer.

Rubinstein, Aynat, Hillary Harner, Elizabeth Krawczyk, Daniel Simonsson, Graham Katz, and Paul Portner. 2013. Toward fine-grained annotation of modality in text. In Proceedings of IWCS 2013 Workshop on Annotation of Modal Meanings in Natural Language (WAMM), pages 38–46. Potsdam, Germany.

Ruppenhofer, Josef and Ines Rehbein. 2012. Yes we can!? Annotating the senses of English modal verbs. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 1538–1545.

Saurí, Roser. 2008. A Factuality Profiler for Eventualities in Text. Ph.D. thesis, Brandeis University.

Saurí, Roser and James Pustejovsky. 2009. FactBank: a corpus annotated with event factuality. Language Resources and Evaluation 43(3):227–268.

Saurí, Roser and James Pustejovsky. 2012. Are You Sure That This Happened? Assessing the Factuality Degree of Events in Text. Computational Linguistics 38(2):262–299.

Spreyer, Kathrin and Anette Frank. 2008. Projection-based Acquisition of a Temporal Labeller. In Proceedings of the 3rd International Joint Conference on Natural Language Processing (IJCNLP ’08). Hyderabad, India.

Szarvas, György, Veronika Vincze, Richard Farkas, György Mora, and Iryna Gurevych. 2012. Cross-Genre and Cross-Domain Detection of Semantic Uncertainty. Computational Linguistics 38(2).

Thompson, Paul, Gilua Venturi, John McNaught, Simonetta Montemagni, and Sophia Ananiadou. 2008. Categorising modality in biomedical texts. In Proceedings of the Sixth International Conference on Language Resources

and Evaluation (LREC-2008).

Tiedemann, Jörg. 2012. Parallel Data, Tools and Interfaces in OPUS. In N. Calzolari, K. Choukri, T. Declerck, M. U. Doğan, B. Maegaard, J. Mariani, J. Odijk, and S. Piperidis, eds., Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 2214–2218. Istanbul, Turkey. ISBN 978-2-9517408-7-7.

Wiebe, Janyce, Theresa Wilson, and Claire Cardie. 2005. Annotating ex- pressions of opinions and emotions in language. Language resources and evaluation 39(2-3):165 – 210.

Yarowsky, David, Grace Ngai, and Richard Wicentowski. 2001. Inducing Multilingual Text Analysis Tools via Robust Projection across Aligned Corpora. In Proceedings of HLT 2001, First International Conference on Human Language Technology Research.

Yimam, Seid Muhie, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013. WebAnno: A Flexible, Web-based and Visually Supported System for Distributed Annotations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (System Demonstrations) (ACL-2013).

Zhou, Mengfei. 2015. Cross-lingual Semi-Supervised Modality Tagging. Bachelor thesis, Department of Computational Linguistics, Heidelberg University.

Zhou, Mengfei, Anette Frank, Annemarie Friedrich, and Alexis Palmer. 2015. Semantically Enriched Models for Modal Sense Classification. In Proceedings of the EMNLP 2015 Workshop LSDSem: Linking Models of Lexical, Sentential and Discourse-level Semantics. Lisbon, Portugal.


  • There are currently no refbacks.