MaskedLMScorer¶

class minicons.scorer.MaskedLMScorer(model: str | torch.nn.Module, device: str | None = 'cpu', tokenizer=None, PLL_metric: str = 'original', **kwargs)¶

Bases: LMScorer

Class for Masked Langauge Models such as BERT, RoBERTa, etc.

Parameters:

model – should be path to a model (.pt or .bin file) stored locally, or name of a pretrained model stored on the Huggingface Model Hub, or a model (torch.nn.Module) that have the same signature as a Huggingface model obtained from AutoModelForMaskedLM. In the last case, a corresponding tokenizer must also be provided.
device (str, optional) – device type that the model should be loaded on, options: cpu or cuda:{0, 1, …} or auto
tokenizer – if provided, use this tokenizer.

add_special_tokens(text: str | Iterable[str]) → List[str]¶

Reformats input text to add special model-dependent tokens.

Parameters:: text (Union[str, List[str]]) – single string or batch of strings to be modified.
Returns:: Modified input, containing special tokens as per tokenizer specification
Return type:: List[str]

mask(sentence_words: Tuple[str, str] | List[Tuple[str, str]]) → Tuple[List[str], List[str]]¶

Processes a list of (sentence, word) into input that has the word masked out of the sentence.

Note: only works for masked LMs.

Parameters:: sentence_words (Union[Tuple[str], List[Tuple[str]]]) – Input consisting of [(sentence, word)], where sentence is an input sentence, and word is a word present in the sentence that will be masked out.
Returns:: Tuple (sentences, words)

cloze(sentence_words: Tuple[str, str] | List[Tuple[str, str]], PLL_metric: str | None = None, probs: bool | None = False) → List[float]¶

Runs inference on masked input. Note: only works for masked LMs.

Parameters:

sentence_words (Union[Tuple[str], List[Tuple[str]]]) – Input consisting of [(sentence, word)], where sentence is an input sentence, and word is a word present in the sentence that will be masked out and inferred.
PLL_metric – PLL scoring strategy to be used. Options: original or within_word_l2r. Default: original For motivation as to why to use within_word_l2r PLL scoring, see Kauf & Ivanova (2023): https://arxiv.org/abs/2305.10588
probs – whether to return probabilities (if True) or log probabilities (if False)

Returns:

A list of tensors corresponding to (log) probabilities for the desired word in context

get_masked_tensors(encoded: transformers.BatchEncoding, PLL_metric: str | None = None, targets_start: List[int] | None = None, targets_end: List[int] | None = None) → Iterable[Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]]¶: Create batches of tokenized instances with each token in the sequence replaced by the mask token.

prepare_text(text: str | List[str] | transformers.BatchEncoding, PLL_metric: str | None = 'original') → Iterable[Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]]¶

Prepares a batch of input text into a format fit to run MLM scoring on.

Borrows preprocessing algorithm from Salazar et al. (2020), and modifies code from the following github repository by simonpri: https://github.com/simonepri/lm-scorer

Parameters:

text – batch of sentences to be prepared for scoring.
PLL_metric – PLL scoring strategy to be used. Options: original or within_word_l2r. Default: original For motivation as to why to use within_word_l2r PLL scoring, see Kauf & Ivanova (2023): https://arxiv.org/abs/2305.10588

Returns:

Batch of formatted input that can be passed to compute_stats

Prepares a batch of input text into a format fit to run LM scoring on.

Borrows preprocessing algorithm from Salazar et al. (2020), and modifies code from the following github repository by simonpri: https://github.com/simonepri/lm-scorer

Parameters:

prefix (Union[str, List[str]]) – Batch of prefixes/prime/preambles on which the LM is conditioned.
stimuli (Union[str, List[str]]) – Batch of continuations that are scored based on the conditioned text (provided in the prefix). The positions of the elements match their counterparts in the prefix.

Returns:

Batch of formatted input that can be passed to compute_stats

distribution(batch: Iterable) → torch.Tensor¶

Returns a distribution over the vocabulary of the model.

Parameters:: batch (Iterable) – A batch of inputs fit to pass to a transformer LM.
Returns:: Tensor consisting of log probabilies over vocab items.

cloze_distribution(queries: Collection[Tuple[str, str]] | Tuple[str, str]) → torch.Tensor¶

Accepts as input batch of [(s_i, bw_i)] where s_i is a prompt with an abstract token (bw_i) representing a blank word and returns a distribution over the vocabulary of the model.

Parameters:: queries (Iterable) – A batch of [(s_i, bw_i)] where s_i is a prompt with an abstract token (bw_i) representing a blank word
Returns:: Tensor contisting of log probabilities over vocab items.

logprobs(batch: Iterable, rank=False) → List[Tuple[torch.Tensor, str]] | List[Tuple[torch.Tensor, str, int]]¶

Deprecated since version Use: compute_stats() instead.

Parameters:

batch (Iterable) – A batch of inputs fit to pass to a transformer LM.
rank (bool) – Specifies whether to also return ranks of words.

Returns:

List of MLM score metrics and tokens.

Return type:

Union[List[Tuple[torch.Tensor, str]], List[Tuple[torch.Tensor, str, int]]]

compute_stats(batch: Iterable, rank: bool = False, base_two: bool = False, prob: bool = False, return_tensors: bool = False) → Tuple[List[float], List[int]] | List[float] | torch.Tensor | Tuple[torch.Tensor, torch.Tensor]¶

Primary computational method that processes a batch of prepared sentences and returns per-token scores for each sentence. By default, returns log-probabilities.

Parameters:

batch (Iterable) – batched input as processed by prepare_text or prime_text.
rank (bool) – whether the model should also return ranks per word (based on the conditional log-probability of the word in context).
prob (bool) – whether the model should return probabilities instead of log-probabilities. Can only be True when base_two is False.
base_two (bool) – whether the base of the log should be 2 (usually preferred when reporting results in bits). Can only be True when prob is False.
return_tensors (bool) – whether the model should return scores as a list of tensors instead of a list of lists. This is important in some other convenient methods used in the package.

Returns:

Either a tuple of lists, each containing probabilities and ranks per token in each sentence passed in the input.

Return type:

Union[Tuple[List[float], List[float]], List[float]]

sequence_score(batch, reduction=<function MaskedLMScorer.<lambda>>, prob: bool = False, base_two: bool = False, **kw)¶

Pooled estimates of sequence log probabilities (or some modification of it).

Parameters:

batch (Union[str, List[str]]) – a batch of sequences whose score you want to calculate.
reduction (Callable) – Reduction function, is selected to be lambda x: x.mean(0).item() by default, which stands for the avg. log-probability per token for each sequence in the batch.
kw – model-specific keyword arguments to pass to the prepare_text function

Returns:

List of floats specifying the desired score for the stimuli part of the input, e.g., P(stimuli | preamble).

Return type:

List[float]

TODO: reduction should be a string, if it’s a function, specify what kind of function. –> how to ensure it is always that type?

token_score(batch: str | List[str], surprisal: bool = False, prob: bool = False, base_two: bool = False, rank: bool = False, PLL_metric: str = 'original') → List[Tuple[str, float]] | List[Tuple[str, float, int]]¶

For every input sentence, returns a list of tuples in the following format:: (token, score),

where score represents the log-probability (by default) of the token given context. Can also return ranks along with scores.

Parameters:

batch (Union[str, List[str]]) – a single sentence or a batch of sentences.
surprisal (bool) – If True, returns per-word surprisals instead of log-probabilities.
prob (bool) – If True, returns per-word probabilities instead of log-probabilities.
base_two (bool) – If True, uses log base 2 instead of natural-log (returns bits of values in case of surprisals)
rank (bool) – If True, also returns the rank of each word in context (based on the log-probability value)
PLL_metric (str) – PLL scoring strategy to be used. Options: original or within_word_l2r. Default: original For motivation as to why to use within_word_l2r PLL scoring, see Kauf & Ivanova (2023): https://arxiv.org/abs/2305.10588

Returns:

A List containing a Tuple consisting of the word, its associated score, and optionally, its rank.

Return type:

Union[List[Tuple[str, float]], List[Tuple[str, float, int]]]

conditional_score(prefix: str | ~typing.List[str], stimuli: str | ~typing.List[str], suffix: None | str | ~typing.List[str] = None, separator: str = ' ', reduction: ~typing.Callable = <function MaskedLMScorer.<lambda>>, prob: bool = False, base_two: bool = False, **kw) → List[float]¶

Pooled estimates of sequence log probabilities (or some modification of it), given a prefix. Pooling is usually done using a function that is passed to the method.

Parameters:

prefix (Union[str, List[str]]) – a batch of prefixes or primes passed to the language model. This is what the sequence is conditioned on, and the model ignores the word probabilities of this part of the input in estimating the overall score.
stimuli (Union[str, List[str]]) – a batch of sequences (same length as prefix) that form the main input consisting of the sequence whose score you want to calculate.
reduction (Callable) – Reduction function, is selected to be lambda x: x.mean(0).item() by default, which stands for the avg. log-probability per token for each sequence in the batch.
kw – model-specific keyword arguments to pass to the prepare_text function

Returns:

List of floats specifying the desired score for the stimuli part of the input, e.g., P(stimuli | preamble).

Return type:

List[float]

partial_score(preamble: str | ~typing.List[str], stimuli: str | ~typing.List[str], suffix: None | str | ~typing.List[str] = None, separator: str = ' ', reduction: ~typing.Callable = <function MaskedLMScorer.<lambda>>, **kwargs) → List[float]¶: Deprecated since version Use: conditional_score() instead.