National Centre for Language Technology

NCLT Links

NCLT Seminar Series

Automatic Grammaticality Judgments

In this talk I will present an evaluation of four approaches to automatic grammaticality judgements. Such judgements can be used to automatically grade essays or to trigger a computationally expensive error analysis. The first approach follows the traditional view that the grammar determines grammaticality. The test corpus is parsed with the XLE parser and "starred" sentences are classified as ungrammatical. The second approach is similar. Here we prune a PTB-trained PCFG so that it rejects ungrammatical input. Thirdly, n-gram methods are considered. If a sentence contains an n-gram below a certain frequency threshold, it is rejected. Finally, my own approach (developed in collaboration with Jennifer Foster) is included in the evaluation. The approach compares the actual probability output of a statistical parser with a probability estimated from a reference corpus of grammatical sentences in order to judge the grammaticality.

View slides

Annotation Schemes and Parser Evaluation for German

A long-standing and unresolved issue in the parsing literature is whether parsing less-configurational languages is harder than e.g. parsing English. German is a case in point. Results from Dubey and Keller (2003) suggest that state-of-the-art parsing scores for German are generally lower than those obtained for English, while recent results from Kuebler et al. (2006) raise the possibility that this might be an artifact of encoding schemes and data structures of treebanks, which serve as training resources for probability parsers. In the talk I present new experiments to test this claim. We use the PARSEVAL metric, the Leaf-Ancestor metric as well as a dependency-based evaluation, and present complimentary approaches measuring the effect of controlled error insertion on treebank trees and parser output. We also provide extensive cross-treebank conversion. The results of the experiments show that, contrary to Kuebler et al. (2006), the question whether or not German is harder to parse than English remains undecided.

View slides

Some Current Trends in Machine Translation

For reasons that I can't quite recall, I've been the track coordinator for EACL-06 and ACL-07. Given that, I have some interesting (I think!) statistics on the trends from one conference to the other regarding topics of papers, and country of origin of those same papers. I'll then finish with some predictions arising from those figures.

View slides

Exploiting Multi-Word Units in Probabilistic Treebank-Based Parsing and Generation

I present the results of several experiments using multi-word units (MWUs) as a means to impose constraints on both probabilistic parsing and surface generation with automatically-acquired (treebank-based) grammars. In the case of surface realisation from LFG f-structures with automatically-acquired treebank-based LFG approximations modest but significant gains in accuracy can be made. Experiments integrating the same MWUs in treebank-based probabilistic parsing yielded smaller, but still statistically significant gains. I analyse the results and offer a number of explanations as to why the gains achieved are smaller than might be naively expected.

Zero pronoun identification in Japanese corpus

I talk about zero pronoun identification in Japanese corpus. Since zero pronouns appear quite often in Japanese texts, identifying them is one of the important issues in Japanese NLP, and is also required in long distance dependency (LDD) resolution at the level of f-structure representation, and in automatic case-frame extraction from a large corpus. I introduce a simple method of zero pronoun identification which uses verbal morphological features which signify transitivity of a verb, along with the probability of the cooccurrence of a verb and nouns which are attached with certain case-marking particles. I will show and analyze the result of applying the method to 500 sentences randomly chosen from Kyoto Text Corpus, and the parsing output of the same sentences by the Japanese dependency parser which does not take zero pronouns into account, and try to explain the advantage and drawback of the method, and possible ways to improve its performance.

View slides

The Link Between Controlled Language & Post-Editing: An Empirical Investigation of Technical, Temporal and Cognitive Effort

Studies on Controlled Language (CL) suggest that by removing certain linguistic features that are known to be problematic for Machine Translation (MT) from a source text, the MT output can be improved. A further assumption is that an improvement in MT output will result in lower post-editing effort. With the ever-increasing emphasis in the translation industry on higher volumes and faster throughput, it is not surprising that this assumption is of interest to those who manage multi-lingual high-volume translation projects. Increasingly, translation service providers are asked to provide post-editing services in addition to their traditional translation/localisation services. The expectation is that post-editing will be faster than human translation and that, therefore, post-editing should not cost as much as translation. However, the assumption that CL reduces post-editing effort has not been tested empirically. It is worthy of closer inspection, not least because CLs can cover a broad range of linguistic features (OBrien 2003). This paper presents results from a study designed to test the assumed link between CL and post-editing effort by measuring the technical, temporal and cognitive post-editing effort (Krings 2001) for English sentences in a user manual that have been translated into German using an MT system and that have been subsequently post-edited by nine professional translators. In this study, the linguistic features known to be problematic for MT are called negative translatability indicators or NTIs for short. The post-editing effort for sentences containing NTIs is compared with the post-editing effort for sentences where all known NTIs have been removed. In addition, relative post-editing effort (Krings 2001) a comparison of post-editing effort and translation effort - is measured. A comparison will be made between NTIs that generate a high-level of post-editing effort and those that generate a lower level of post-editing effort. The methodologies employed include the use of the keyboard monitoring tool, Translog (Jakobsen 1999, Hansen 2002), and Choice Network Analysis (Campbell 1999).

Campbell, Stuart (1999), A Cognitive Approach to Source Text Difficulty in Translation, in Target, 11:1, pp. 33-63.

Hansen, Gyde (ed) (2002), Empirical Translation Studies Process and Product, Copenhagen Studies in Language 27, Copenhagen: Samfundslitteratur.

Jakobsen, Arnt Lykke (1999), Logging Target Text Production with Translog, in Hansen, Gyde (ed), Probing the Process in Translation: Methods and Results, Copenhagen Studies in Language 24, Copenhagen: Samfundslitteratur, pp. 9-20.

Krings, Hans P. (2001), Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes, Kent/Ohio: Kent State University Press, edited/translated by G. S. Koby et al.

OBrien, Sharon (2003), Controlling Controlled English An Analysis of Several Controlled Language Rule Sets, in Proceedings of EAMT/CLAW 2003, Dublin: Dublin City University, pp. 105-114.

View slides

Robust Language Pair-Independent Sub-Tree Alignment

Data-driven approaches to machine translation (MT) achieve state-of-the-art results. Many syntax-aware approaches, such as Example-Based Machine Translation and Data-Oriented Translation, make use of tree pairs aligned at sub-sentential level. Obtaining sub-sentential alignments manually is time-consuming and error-prone, and requires expert knowledge of both source and target languages. We propose a novel, language pair-independent algorithm which automatically induces alignments between phrase-structure trees. We evaluate the alignments themselves against a manually aligned gold standard, and perform an extrinsic evaluation by using the aligned data to train and test a DOT system. Our results show that translation accuracy is comparable to that of the same translation system trained on manually aligned data, and coverage improves.

View slides

Language Modeling by Random Forests

Automatic Speech Recognition is based on several components: signal processor, acoustic model, language model, and search. In this talk, we explore the use of Random Forests (RFs) in language modeling, the problem of predicting the next word based on words already seen. The goal is to develop a new language model smoothing technique based on randomly grown Decision Trees (DTs). This new technique is complementary to many of the existing techniques dealing with data sparseness.
Random forests were studied by Breiman in the context of classification into a relatively small number of classes. We study their application to n-gram language modeling which could be thought of as classification into a very large number of classes. Unlike regular n-gram language models, RF language models have the potential to generalize well to unseen data, even when histories are long (>4). We show that our RF language models are superior to regular n-gram language models in reducing both the entropy and the word error rate in a large vocabulary speech recognizer.

Estimating Importance Features for Fact Mining
(With a Case Study in Biography Mining)

We present a transparent model for ranking sentences that incorporates topic relevance as well as an aboutness and importance feature. We describe and compare five methods for estimating the importance feature. The two key features that we use are graph-based ranking and ranking based on reference corpora of sentences known to be important. Independently those features do not improve over the baseline, but combined they do. While our experimental evaluation focuses on informational queries about people, our importance estimation methods are completely general and can be applied to any topic.

Show Slides

How useful is machine-translated technical documentation? Let's ask users!

Previous studies suggest that the application of Controlled Language (CL) rules can significantly improve the readability, consistency, and machine-translatability of technical documentation. One of the justifications for the application of CL rules is that they can reduce the post-editing effort required to bring Machine Translation (MT) output to acceptable quality. In certain situations, however, post-editing services may not always be a viable solution. Web-based information is often expected to be made available in real-time to ensure that its access is not restricted to certain users based on their locale. Uncertainties remain with regard to the actual usefulness of MT output for such users, as no empirical study has examined the impact of CL rules on the usefulness and comprehensibility of MT technical documents from a Web user's perspective. This presentation focuses on the results of an online experiment conducted at Symantec, a leader in Internet security technology. Using a customer satisfaction questionnaire, a set of machine-translated technical support documents was published and randomly evaluated by genuine French and German users. The findings indicate that the introduction of CL rules can have a significant impact on the comprehensibility of German MT documents.

Show Slides

Using F-structures in Automatic Machine Translation Evaluation

A feature valuation approach to the prohibition on two definite determiners in genitive noun phrases in Irish

The objective of this talk is to explain the prohibition on two determiners in genitive noun phrases in Irish using the frameworks of the Minimalist Program and Distributed Morphology. I will first give a brief overview of Generative Syntax, the Minimalist Program and Distributed Morphology. This will be followed with a recap of previous work on Irish noun phrases involving the DP Hypothesis. I will then introduce the notion of feature valuation in Distributed Morphology which includes a particular view of nominalisation. These concepts provide the framework for an elegant explanation of Determiner-Noun agreement, Genitive case assignment and Definiteness agreement. The prohibition on two determiners in genitive noun phrases in Irish follows naturally from this explanation.

Show Slides

The schedule of presenters for the 2006/2007 series (Semester 1) is as follows:

Exploring Interfaces and Issues in Intelligent Computer-Aided Language Learning

While foreign language teaching since the 60s has focused on communication and culture, a growing body of research since the 90s shows that awareness of language categories, forms and rules is an important component for an adult learner to successfully acquire a foreign language. Such research can be viewed as providing a sound basis for developing and integrating intelligent computer-aided language learning (ICALL) tools that foster learner awareness of language forms and categories, based on natural language processing identifying the relevant linguistic properties. At the same time, there are virtually no such ICALL tools being used in real-life foreign language teaching today, and most ICALL research has focused on issues disconnected from second language acquisition insights and foreign language teaching needs. The situation is a clear opportunity for an interdisciplinary approach combining linguistic modeling, learner/cognitive modeling, and instruction/activity modeling to develop ICALL applications for real-life language teaching. The talk will discuss some of these interfaces that we are working on in the OSU ICALL group in the context of our work on the TAGARELA and the WERTi system, as well as the issues involved in dealing with word order errors and possibly also the work on shallow semantic analysis of student responses we have recently started on.

Using Web Data for Linguistic Purposes

Corpora have become a widely accepted tool in all areas of linguistics during the last decades. But with increasing acceptance there is also a growing demand for more and larger corpora, backed by the progress in computer technology, especially in computing power and storage. While the creation of corpora is a timeconsuming and costly process, there is a large amount of data freely available on the web, only waiting for the linguist. But how can we use this huge amount of data? What are the requirements which have to be fulfilled, which rules have to be followed in order to get valid results from web data? The talk will give an overview on general requirements and principles of corpus annotation and, following the paper by Lüdeling, Evert, and Baroni (to appear), focus on the specific characteristics of web data.

View slides

TAG - Tree-Adjoining Grammars

Tree-adjoining grammar (TAG) is a grammar formalism defined by Aravind Joshi [Joshi & Takahashi, 1975]. The talk will introduce the basic theories of TAG formalism which provides an extended domain of locality. This extended domain is achieved by specifying the building blocks as structured objects (elementary trees) rather than strings and two universal combining operations (substitution & adjoining). Using lexicalized elementary structured objects it is possible to study directly many aspects of strong generative capacity which are more relevant to the linguistic description. The talk will also skim over some applications based on TAG, including development (hand-crafted & automatic extraction) and parsing of TAGs.

WiQA 2006: Question Answering using Wikipedia

In this presentation, I will mainly focus on my recent activity, which is developing a system for our participation in the WiQA 2006 pilot task: question answering using Wikipedia, free online encyclopedia. At the end of my presentation, I will briefly summarize my previous experience in other areas of natural language processing such as machine translation, parsing and information extraction.

The WiQA 2006 pilot deals with access to the content of Wikipedia. At WiQA, information access is considered both from a reader's point of view and from an author's point of view. WiQA derives much of its motivation from the observation that, in the Wikipedia context, the distinction between reader and author has become blurred. The overlap in the roles of the different user types of Wikipedia motivates new modes of information access, one that can support the emerging dual roles identified above.

WiQA 2006 attempts to address this issue by formulating the task definition as follows. Given a topic (and associated article) in one language, identify relevant snippets on this topic from other articles in the same language or even in other languages. In the context of Wikipedia, having a system that offers this type of capability is important both for using Wikipedia as a reader and as an author. It provides effective access to additional relevant information in Wikipedia that is not found in the main article on the topic, which, we believe, is important for both use cases. In our system, we implemented a simple but effective importance estimation method that combines a corpus-based approach to capturing the knowledge encoded in sentences known to be important with a graph-based method for ranking sentences.

Automatic conversion of a Japanese text corpus into f-structures

The talk will introduce an automatic f-structure conversion algorithm for Kyoto Text Corpus (KTC). KTC is a dependency-based, non CFG-based text corpus of about 40,000 sentences taken from a Japanese newspaper. The talk will introduce some issues in Japanese grammar (e.g., free word order, subject and object ellipses, use of particles to specify the grammatical function of a noun) which are relevant to the automatic conversion of KTC into f-structure. In this conversion algorithm, each unit (a content word plus a particle following the word) is converted into one f-structure, and all the units in one sentence are unified into the f-structure of the sentence according to the dependency among these units. The talk will also deal with some problems in this algorithm and possible solutions for them employing the L/R context among units.

Unsupervised Multilingual Sentence Boundary Detection

I suggest that people read up to page 8, have a look at the headlines on pages 11 to 14, read section 6.3 on page 18/19, look at tables on pages 19 to 22, tables 14 and 15 on page 23/24, and at one of the other systems "Punkt" is compared to, for example "Satz" on page 32.

Vie w paper     Kiss & Strunk (2005)

View slides

Grammar Inference, Automata Induction, and Language Acquisition

The natural language learning problem has attracted the attention of researchers for several decades. Computational and formal models of language acquisition have provided some preliminary, yet promising insights [into] how children learn the language of their community. Further, these formal models also provide an operational framework for the numerous practical applications of formal language learning. (Parekh and Honavar, 2000)." This NCLT seminar will introduce the notion of formal grammatical inference. We will then discuss one of the several algorithms presented in Parekh and Honavar (2000) for regular language inference, and some associated results which motivated the creation of this algorithm. Finally, we will look into the modification of the discussed algorithm for deriving simplified input for and thereby acquiring a generalized simplification of an f-structure automatic annotation scheme.

Machine Learning for Structured Prediction

This talk will offer an overview of discriminative learning techniques for structured prediction task in Natural Language Processing. Most tasks in NLP involve predicting output which are structured in some way: they can be sequences, syntactic trees, dependency graphs etc. Most common Machine Learning algorithms (e.g.. k-nearest neighbours, perceptrons, logistic regression, Support Vector Machines) deal with the classification setting where outputs are label taken from a finite set. We present an overview of Machine Learning techniques which extend and build upon classification algorithms, and which can be used to solve the more challenging task of generating complex outputs.

View slides

Toward a Syntax-rich Statistical Machine Translation

To add more syntactic information into statistical machine translation (SMT) has become a common sense in the community of machine translation. In this talk, we will start with a brief introduction to the well-established, most prevailing phrase-based SMT models. After an analysis of the weakness of phrase-based SMT, a few impressive syntax-based SMT systems which have immerged in NIST 2006 and IWSLT 2006 will be introduced. Factored phrase-based SMT (moses), Hierarchical phrase-based SMT (formal syntax), Tree-to-String syntax-based SMT and String-to-Tree syntax-based SMT(linguistic syntax) , as four different SMT models which have outperformed the state-of-the-art phrase-based SMT systems will be introduced. We will give a detailed explanation, including training and decoding, to hierarchical phrase-based SMT and String-to-Tree syntax-based SMT. In the end, we will talk about the recent progress in our machine translation system---MaTrex.

View slides

Grammatical Machine Translation

In "Grammatical Machine Translation" by Stefan Riezler and John Maxwell the authors present an approach to statistical machine translation that combines ideas from phrase-base SMT and traditional grammar-based MT. The system described in the paper incorporates the concept of multi-word translation units into the transfer of dependency structure snippets. In addition, it models and trains statistical components according to state of the art SMT systems. Target dependency structure snippets are input to a grammar-based generator. An experimental evaluation detailed in the paper showed that this incorporation of a grammar based generator into SMT framework provides improved grammaticality while achieving state-of-the-art quality for in-coverage examples, suggesting a possible hybrid framework. In this talk I will present this paper.

View paper

View slides

Deduction, Induction, Abduction, and Analogy : a review of the main Inference Paradigms and their links to Natural Language Processing

Natural Language Processing is about making decisions in various contexts (phonological, morphological, syntactic, semantic, translational, etc.). These decisions are usually based on some inferential process that motivates the choices that are made. The characterization of inferential processes have been studied in particular in the field of Artificial Intelligence, with the goal to be able to reproduce human decisions. In this talk, we will review the main inference paradigms: deduction (going from the general to the particular), induction (going from the particular to the general), analogy (going from the particular to the particular), and abduction, and highlight their relationships with NLP.

View slides

Comparing Parser Evaluation Schemes

This seminar will be based on selected papers from the Proceedings of the ``Beyond Parseval - Towards Improved Evaluation Measures for Parsing Systems" Workshop at LREC-2002. The widely known Parseval metrics for evaluating parser accuracy will be presented and then compared to proposed alternatives - other constituency-based metrics such as the Leaf-Ancestor metric and evaluation schemes based on dependency relations.

View slides

Dependency-Based Automatic Evaluation for Machine Translation

In this paper we present a novel method for evaluating the output of Machine Translation (MT), based on comparing the dependency structures of the translation and reference rather than their surface string forms. Our method uses a treebank-based, wide coverage, probabilistic Lexical-Functional Grammar (LFG) parser to produce a set of structural dependencies for each translation-reference sentence pair, and then calculates the precision and recall for these dependencies. Our dependency-based evaluation, in contrast to most popular string-based evaluation metrics, will not unfairly penalize perfectly valid syntactic variations in the translation. In addition to allowing for legitimate syntactic differences, we use paraphrases in the evaluation process to account for lexical variation. In comparison with other metrics on 16,800 sentences of Chinese-English newswire text, our method reaches high correlation with human scores. An experiment with two translations of 4,000 sentences from Spanish-English Europarl shows that, in contrast to most other metrics, our method does not display a high bias towards statistical models of translation.

View slides

Data-Driven Sign Language MT

Just as spoken language MT is moving towards data-driven approaches, Sign Language (SL) MT is following suit. In this presentation I will briefly overview my SL MT work to date and include recent evaluation results and advancements. Pending the visit from Daniel Stein of RWTH Aachen University, Germany, I will introduce his Statistical MT work on German Sign Language and discuss the collaboration work we are currently undertaking on SL->spoken language MT and our intentions to develop a SL<->SL translation system.

Statistical Methods for Sign Language Translation

Being a language with very little training data available, the translation of sign languages is a very challenging task. Moreover, the specific visual modality of a sign language introduces additional problems. In my talk, I will give a brief overview of the experiences made at the RWTH Aachen University, as well as talk about ongoing and future projects and ideas.

Details of previous seminars:
Seminar Series 2005/06
Seminar Series 2004/05
Seminar Series 2003/04
Seminar Series 2002/03
Seminar Series 2001/02

Last Updated: 16^st November 2006 by aclweb@computing.dcu.ie