Marie-Catherine de Marneffe’s research while affiliated with The Ohio State University and other places

... With good word embeddings, larger nouns are expected to have higher projection scores. The validity of the adjective scales from word representations is shown by Kim and de Marneffe (2013). ...

Reference:

Visual Commonsense in Pretrained Unimodal and Multimodal Models

... A study with a similar aim is presented by Chen et al. (2024). They explore whether the softmax probability distributions solicited by a few explanations from two LLMs (Mixtral and Llama 3) can approximate human judgement distributions (HJD) on the ChaosNLI and VariErr NLI (Weber-Genzel et al., 2024) benchmarks. Similarly, Baan et al. (2024) and compare human and model judgement distributions on ChaosNLI, finding that models fail to capture human distributions in LLMs and model confidences. ...

Reference:

Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models

... To the best of our knowledge, no such dataset exists yet. (ii) A new methodology to detect errors: we collect multiple annotations, where each label comes with an ecologically valid explanation inspired by Jiang et al. (2023), and propose to pair them with validity judgments to identify errors. (iii) Finally, we benchmark existing AED methods and GPTs in a challenging setup, where the task is to tease apart error from plausible human label variation. ...

Reference:

VariErr NLI: Separating Annotation Error from Human Label Variation

... This is likely because the additional selection and ranking steps implemented by Whistely et al. (2022) and the lack thereof shown within the LS system provided by North et al. (2022a) (Section 3.2). Wilkens et al. (2022) likewise experimented with a range of monolingual transformers for SG. They employed an ensemble of BERT-like models with three distinct masking strategies: 1) copy, 2) query expansion, and 3) paraphrase. ...

Reference:

Deep learning approaches to lexical simplification: A survey

... Santy et al. (2023) and Forbes et al. (2020), explore annotator disagreement in safety, looking specifically at how morality and toxicity judgments vary across users of different backgrounds. Prior works have analyzed disagreements in NLI (Pavlick & Kwiatkowski, 2019;Liu et al., 2023), and Jiang & Marneffe (2022) develop an NLI-specific taxonomy of disagreement causes. Sandri et al. (2023) similarly explores annotator disagreements in toxicity detection, and develop a taxonomy of disagreement causes for their task. ...

Reference:

Diverging Preferences: When do Annotators Disagree and do Models Know?

... Generated Text Detection The burgeoning progress in the generation capabilities of large language models has led to a corresponding increase in research and development efforts in the field of detection. Several recent efforts look at methods, varying from simple feature-based classifiers to fine-tuned language model-based detectors, in order to classify whether a piece of input text is human-written or AI-generated (Ippolito et al., 2019;Gehrmann et al., 2019;Mitchell et al., 2023), along with methods that specifically focus on AI-generated news (Zellers et al., 2019;Bogaert et al., 2022;Kumarage et al., 2023). A related direction of work is that of authorship attribution (AA). ...

Reference:

ConDA: Contrastive Domain Adaptation for AI-generated Text Detection

... We frame belief prediction as a regression task, following previous literature [9,17,18,19]. Given an utterance and a sequence of corresponding features S = [f1, f2, ..., fn], we wish to produce a valueŷ ∈ [−3, 3] that is as close to the annotation value y as possible. ...

Reference:

Multimodal Belief Prediction

... In other words, the world is not just black and white. Human label variation (HLV, as termed by Plank 2022) has been shown on a wide range of NLP tasks (de Marneffe et al., 2012;Plank et al., 2014;Aroyo and Welty, 2015), including in natural language inference (NLI; Pavlick and Kwiatkowski 2019; Zhang and de Marneffe 2021). NLI involves determining whether a hypothesis is true (Entailment), false (Contradiction), or neither (Neutral), assuming the truth of a given premise; see Figure 1 for an example with plausible labels. ...

Reference:

VariErr NLI: Separating Annotation Error from Human Label Variation

... Essa tarefa está compreendida dentro do escopo conhecido como enhanced dependencies (ED). As ED foram desenvolvidas primeiro no inglês (Schuster & Manning, 2016), depois foram generalizadas para outras línguas pela abordagem Universal Dependencies (UD) 1 (Nivre et al., 2018, de Marneffe et al. 2021) e instanciadas para o português (Pagano et al., 2023). ...

Reference:

Inferências baseadas em sintaxe: a anotação de sujeitos implícitos

... Yeomans et al. [174] develop a package in R that provides functions to extract politeness markers in the English language, graphically compare these markers to covariates of interest, develop a supervised model to identify politeness in new documents and inspect high-and low-politeness documents. Aljanaideh et al. [3] model the problem of politeness detection in natural language requests as a clustering task. The authors have created a set of clusters for every word in the request. ...

Reference:

Computational Politeness in Natural Language Processing: A Survey