ACL 2018 Highlights: Understanding Representations

This submit discusses highlights of the 56th Annual Assembly of the Affiliation for Computational Linguistics (ACL 2018).

This submit initially appeared on the AYLIEN weblog.

I attended the 56th Annual Assembly of the Affiliation for Computational Linguistics (ACL 2018) in Melbourne, Australia from July 15-20, 2018 and introduced three papers . It’s foolhardy to attempt to condense a whole convention into one matter; nevertheless, looking back, sure themes seem notably pronounced. In 2015 and 2016, NLP conferences had been dominated by phrase embeddings and a few individuals had been musing that Embedding Strategies in Pure Language Processing was a extra acceptable title for the Convention on Empirical Strategies in Pure Language Processing, one of many prime conferences within the subject.

In response to Chris Manning, 2017 was the 12 months of the BiLSTM with consideration. Whereas BiLSTMs optionally with consideration are nonetheless ubiquitous, the primary themes of this convention for me had been to acquire a greater understanding what the representations of such fashions seize and to reveal them to tougher settings. In my assessment, I’ll primarily give attention to contributions that contact on these themes however may also focus on different themes that I discovered of curiosity.

Probing fashions

It was very refreshing to see that reasonably than introducing ever shinier new fashions, many papers methodically investigated current fashions and what they seize. This was mostly finished by robotically making a dataset that focuses on one specific facet of the generalization behaviour and evaluating totally different educated fashions on this dataset:

• Conneau et al. as an example consider totally different sentence embedding strategies on ten datasets designed to seize sure linguistic options, similar to predicting the size of a sentence, recovering phrase content material, sensitivity to bigram shift, and so on. They discover that totally different encoder architectures can lead to embeddings with totally different traits and that bag-of-embeddings is surprisingly good at capturing sentence-level properties, amongst different outcomes.
• Zhu et al. consider sentence embeddings by observing the change in similarity of generated triplets of sentences that differ in a sure semantic or syntactic facet. They find—among different things—that SkipThought and InferSent can distinguish negation from synonymy, whereas InferSent is best at figuring out semantic equivalence and coping with quantifiers.
• Pezzelle et al. focus particularly on quantifiers and take a look at totally different CNN and LSTM fashions on their means to foretell quantifiers in single-sentence and multi-sentence contexts. They discover that in single-sentence context, fashions outperform people, whereas people are barely higher in multi-sentence contexts.
• Kuncoro et al. consider LSTMs on modeling subject-verb settlement. They discover that with sufficient capability, LSTMs can mannequin subject-verb settlement, however that extra syntax-sensitive fashions similar to recurrent neural community grammars do even higher.
• Blevins et al. consider fashions pretrained on totally different duties whether or not they seize a hierarchical notion of syntax. Particularly, they prepare the fashions to foretell POS tags in addition to constituent labels at totally different depths of a parse tree. They discover that each one fashions certainly encode a major quantity of syntax and—in particular—that language fashions study some syntax.
• Khandelwal et al. present that LSTM language fashions use about 200 tokens of context on common. Phrase order is barely related inside the newest sentence.
• One other attention-grabbing outcome relating to the generalization means of language fashions is because of Lau et al. who discover {that a} language mannequin educated on a sonnet corpus captures meter implicitly at human-level efficiency.
• Language fashions, nevertheless, even have their limitations. Spithourakis and Riedel observe that language fashions are dangerous at modelling numerals and suggest a number of methods to enhance them.
• Liu et al. present that LSTMs educated on pure language knowledge are capable of recall tokens from for much longer sequence than fashions educated on non-language knowledge on the Repl4NLP workshop.

Specifically, I believe higher understanding what data LSTMs and language fashions will grow to be extra vital, as they appear to be a key driver of progress in NLP going ahead, as evidenced by our ACL paper on language mannequin fine-tuning and associated approaches.

Understanding state-of-the-art fashions

Whereas the above research attempt to perceive a selected facet of the generalization means of a specific mannequin class, a number of papers give attention to higher understanding state-of-the-art fashions for a specific activity:

• Glockner et al. centered on the duty of pure language inference. They created a dataset with sentences that differ by at most one phrase from sentences within the coaching knowledge in an effort to probe if fashions can cope with easy lexical inferences. They discover that present state-of-the-art fashions fail to seize many easy inferences.
• Mudrakarta et al. analyse state-of-the-art QA fashions throughout totally different modalities and discover that the fashions typically ignore key query phrases. They then perturb inquiries to craft adversarial examples that considerably decrease fashions’ accuracy.

One other technique to acquire a greater understanding of a mannequin is to research its inductive bias. The Workshop on Relevance of Linguistic Construction in Neural Architectures for NLP (RELNLP) sought to discover how helpful it’s to include linguistic construction into our fashions. One of many key factors of Chris Dyer’s discuss in the course of the workshop was whether or not RNNs have a helpful inductive bias for NLP. Specifically, he argued that there are a number of items of proof indicating that RNNs favor sequential recency, specifically:

1. Gradients grow to be attenuated throughout time. LSTMs or GRUs could assist with this, however additionally they neglect.
2. Individuals have used coaching regimes like reversing the enter sequence for machine translation.
3. Individuals have used enhancements like consideration to have direct connections again in time.
4. For modeling subject-verb settlement, the error price will increase with the variety of attractors.

In response to Chomsky, sequential recency will not be the fitting bias for studying human language. RNNs thus do not appear to have the fitting bias for modeling language, which in observe can result in statistical inefficiency and poor generalization behaviour. Recurrent neural community grammars, a category of fashions that generates each a tree and a sequence sequentially by compressing a sentence into its constituents, as an alternative have a bias for syntactic (reasonably than sequential) recency.

Higher understanding representations was additionally a theme on the Illustration Studying for NLP workshop. Throughout his discuss, Yoav Goldberg detailed among the efforts of his group to higher perceive representations of RNNs. Specifically, he mentioned latest work on extracting a finite state automaton from an RNN in an effort to higher perceive what the mannequin has realized. He additionally reminded the viewers that LSTM representations, despite the fact that they’ve been educated on one activity, should not task-specific. They’re typically predictive of unintended elements similar to demographics within the knowledge. Even when a mannequin has been educated utilizing a domain-adversarial loss to provide representations which are invariant of a sure facet, the representations can be nonetheless barely predictive of stated attribute. It will possibly thus be a problem to utterly take away undesirable data from encoded language knowledge and even seemingly excellent LSTM fashions could have hidden failure modes.

A theme that’s intently interlinked with gaining a greater understanding of the restrictions of state-of-the-art fashions is to suggest methods how they are often improved. Specifically, much like adversarial instance paper talked about above, a number of papers tried to make fashions extra sturdy to adversarial examples:

• Cheng et al.
 suggest to make each the encoder and decoder in NMT fashions extra sturdy in opposition to enter perturbations.
• Ebrahimi et al. suggest white-box adversarial examples to trick a character-level neural classifier by swapping few tokens.
• Ribeiro et al. enhance upon the earlier methodology with semantic-preserving perturbations that induce adjustments within the mannequin’s predictions, which they generalize to guidelines that induce adversaries on many situations.
• Bose et al. incorporate adversarial examples into noise contrastive estimation utilizing an adversarially realized sampler. The sampler finds tougher unfavourable examples, which forces the mannequin to study higher representations.

Tim Baldwin mentioned alternative ways to make fashions extra sturdy to a site shift throughout his discuss on the RepL4NLP workshop. The slides will be discovered right here. For utilizing a single supply area, he mentioned a technique to linguistically perturb coaching situations based mostly on several types of syntactic and semantic noise. Within the setting with a number of supply domains, he proposed to coach an adversarial mannequin on the supply domains. Lastly, he mentioned a technique that enables to study sturdy and privacy-preserving textual content representations.

• Finegan-Dollak et al. establish limitations and suggest enhancements to present evaluations of text-to-SQL techniques. They present that the present train-test break up and observe of anonymization of variables are flawed and launch standardized and improved variations of seven datasets to mitigate these.
• Dror et al. give attention to a observe that needs to be commonplace, however is commonly not finished or finished poorly: statistical significance testing. Specifically, they survey latest empirical papers in ACL and TACL 2017 discovering that statistical significance testing is commonly ignored or misused and suggest a easy protocol for statistical significance take a look at choice for NLP duties.
• Chaganty et al. examine the bias of automated metrics similar to BLEU and ROUGE and discover that even an unbiased estimator solely achieves a relatively low error discount. This highlights the necessity to enhance each the correlation of automated metric in addition to cut back the variance of human annotation.

• Shen et al. systematically examine easy phrase embedding-based strategies with pooling to extra complicated fashions similar to LSTMs and CNNs. They discover that for many datasets, phrase embedding-based strategies exhibit aggressive and even superior efficiency.
• Ethayarajh proposes a powerful baseline for sentence embedding fashions on the RepL4NLP workshop.
• In an analogous vein, Ruder and Plank discover that traditional bootstrapping algorithms similar to tri-training make for robust baselines for semi-supervised studying and even outperform latest state-of-the-art strategies.

To be able to consider below such settings, tougher datasets should be created. Yejin Choi argued in the course of the RepL4NLP panel dialogue (a abstract will be discovered right here) that the group pays a variety of consideration to simpler duties similar to SQuAD or bAbI, that are near solved. Yoav Goldberg even went as far as to say that “SQuAD is the MNIST of NLP”. As a substitute, we should always give attention to fixing tougher duties and develop extra datasets with rising ranges of issue. If a dataset is just too onerous, individuals do not work on it. Specifically, the group mustn’t work on datasets for too lengthy as datasets are getting solved very quick nowadays; creating novel and tougher datasets is thus much more vital. Two datasets that search to transcend SQuAD for studying comprehension had been introduced on the convention:

• QAngaroo focuses on studying comprehension that requires to assemble a number of items of knowledge by way of a number of steps of inference.
• NarrativeQA requires perceive of an underlying narrative by asking the reader to reply questions on tales by studying total books or film scripts.

Richard Socher additionally burdened the significance of coaching and evaluating a mannequin throughout a number of duties throughout his discuss in the course of the Machine Studying for Query Answering workshop. Specifically, he argues that NLP requires many sorts of reasoning, e.g. logical, linguistic, emotional, and so on., which can’t all be glad by a single activity.

Analysis on a number of and low-resource languages

One other side of that is to judge our fashions on a number of languages. Emily Bender surveyed 50 NAACL 2018 papers in her discuss talked about above and located that 42 papers consider on an unnamed thriller language (i.e. English). She emphasizes that it is very important title the language you’re employed on as languages have totally different linguistic constructions; not mentioning the language obfuscates this reality.

If our strategies are designed to be cross-lingual, then we should always moreover consider them on the tougher setting of low-resource languages. As an illustration, each of the next two papers observe that present strategies for unsupervised bilingual dictionary strategies fail if the goal language is dissimilar to language similar to with Estonian or Finnish:

• Søgaard et al. probe the restrictions of present strategies additional and spotlight that such strategies additionally fail when embeddings are educated on totally different domains or utilizing totally different algorithms. They lastly suggest a metric to quantify the potential of such strategies.
• Artetxe et al. suggest a brand new unsupervised self-training methodology that employs a greater initialization to steer the optimization course of and is especially sturdy for dissimilar language pairs.

• Dror et al. suggest to make use of orthographic options for bilingual lexicon induction. Although these principally assist for associated languages, additionally they consider on the dissimilar language pair English-Finnish.
• Ren et al. lastly suggest to leverage one other wealthy language for translation right into a resource-poor language
. They discover that their mannequin considerably improves the interpretation high quality of uncommon languages.
• Currey and Heafield suggest an unsupervised tree-to-sequence mannequin for NMT by adapting the Gumbel tree-LSTM. Their mannequin proves notably helpful for low-resource languages.

One other theme in the course of the convention for me was that the sector is visibly making progress. Marti Hearst, president of the ACL, echoed this sentiment throughout her presidential deal with. She used to show what our fashions can and might’t do utilizing the instance of Stanley Kubrick’s HAL 9000 (seen under). Lately, this has grow to be a much less helpful train as our fashions have realized to carry out duties that appeared beforehand a long time away similar to recognizing and producing human speech or lipreading[1]. Naturally, we’re nonetheless far-off from duties that require deep language understanding and reasoning similar to having an argument; nonetheless, this progress is outstanding.

Marti additionally paraphrased NLP and IR pioneer Karen Spärck Jones saying that analysis will not be going round in circles, however climbing a spiral or—maybe extra fittingly—different staircases that aren’t essentially linked however go in the identical course. She additionally expressed a sentiment that appears to resonate with lots of people: Within the Eighties and 90s, with just a few papers to learn, it was undoubtedly simpler to maintain observe of the cutting-edge. To make this simpler, I’ve not too long ago created a doc to gather the cutting-edge throughout totally different NLP duties.

With the group rising, she inspired individuals to take part and volunteer and introduced an ACL Distinguished Service Award for essentially the most devoted members. ACL 2018 additionally noticed the launch (after EACL in 1982 and NAACL in 2000) of its third chapter, AACL, the Asia-Pacific Chapter of the Affiliation for Computational Linguistics.

The enterprise assembly in the course of the convention centered on measures to handle a specific problem of the rising group: the escalating variety of submissions and the necessity for extra reviewers. We will anticipate to see new efforts to cope with the massive variety of submissions on the conferences subsequent 12 months.

Again in 2016, it appeared as if reinforcement studying (RL) was discovering its footing in NLP and being utilized to increasingly duties. As of late, plainly the dynamic nature of RL makes it most helpful for duties that intrinsically have some temporal dependency similar to choosing knowledge throughout coaching[1][1] and modelling dialogue, whereas supervised studying appears to be higher fitted to most different duties. One other vital utility of RL is to optimize the top metric similar to ROUGE or BLEU straight as an alternative of optimizing a surrogate loss similar to cross-entropy. Profitable functions of this are summarization[1][1] and machine translation[1].

Inverse reinforcement studying will be useful in settings the place the reward is just too complicated to be specified. A profitable utility of that is visible storytelling[1]. RL is especially promising for sequential determination making issues in NLP similar to taking part in text-based video games, navigating webpages, and finishing duties. The Deep Reinforcement Studying for NLP tutorial supplied a complete overview of the area.

There have been different nice tutorials as properly. I notably loved the Variational Inference and Deep Generative Fashions tutorial. The tutorials on Semantic Parsing and about “100 stuff you all the time wished to learn about semantics & pragmatics” additionally appeared actually worthwhile. An entire listing of the tutorials will be discovered right here.

Cowl picture: View from the convention venue.

Because of Isabelle Augenstein for some paper options.