A Comparative Evaluation of Deep and Shallow Approaches to the Automatic Detection of Common Grammatical Errors

Joachim Wagner, Jennifer Foster and Josef van Genabith (to appear): A Comparative Evaluation of Deep and Shallow Approaches to the Automatic Detection of Common Grammatical Errors. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) , Prague, June 28-30, 2007

Abstract

This paper compares a deep and a shallow processing approach to the problem of classifying a sentence as grammatically well-formed or ill-formed. The deep processing approach uses the XLE LFG parser and English grammar: two versions are presented, one which uses the XLE directly to perform the classification, and another one which uses a decision tree trained on features consisting of the XLE's output statistics. The shallow processing approach predicts grammaticality based on n-gram frequency statistics: we present two versions, one which uses frequency thresholds and one which uses a decision tree trained on the frequencies of the rarest n-grams in the input sentence. We find that the use of a decision tree improves on the basic approach only for the deep parser-based approach. We also show that combining both the shallow and deep decision tree features is effective. Our evaluation is carried out using a large test set of grammatical and ungrammatical sentences. The ungrammatical test set is generated automatically by inserting grammatical errors into well-formed BNC sentences.

Error Corpus

The error corpus is automatically generated from BNC sentences. We currently work on an improved version of our error creation script. Please contact us if you want to run it on your own version of the BNC (or any other corpus).

Feature Vectors

For many experiments, the features extracted from the sentences and parser output will suffice. The files are available in two formats:

Please contact us if you want to use these files.

Additional Results

We have result tables broken down by sentence length, error type and error subtype. Please contact us if you want to have a look. Maybe we will put them up on this website as well. Some additional diagrams are available in the extra slides of the presentation (after the "Thank You!" slide) and in the ParGram presentation.


<< back
2007-08-07T14:18:26+0100 Tue Aug 7 14:18:26 IST 2007
© 2007 Joachim Wagner jWAGnEr@COMPUTiNg.dcu.IE