VLMScorer¶

class minicons.scorer.VLMScorer(model, device, tokenizer=None, causallm=False, **kwargs)¶

Bases: LMScorer

encode(text: str | List[str], image=None) → transformers.BatchEncoding¶

Encode a batch of sentences using the model’s tokenizer. Equivalent of calling model.tokenizer(input)

Parameters:

text (Union[str, List[str]]) – Input batch/sentence to be encoded.
manual_special (bool) – Specification of whether special tokens will be manually encoded.
return_tensors (str) – returned tensor format. Default ‘pt’

Returns:

Encoded batch

Return type:

BatchEncoding

decode(idx: List[int])¶

Decode input ids using the model’s tokenizer.

Parameters:: idx (List[int]) – List of ids.
Returns:: Decoded strings
Return type:: List[str]

prepare_text(text: str | List[str] | transformers.BatchEncoding, image=None) → Tuple¶

Prepares a batch of input text into a format fit to run LM scoring on.

Parameters:: text – batch of sentences to be prepared for scoring.
Returns:: Batch of formatted input that can be passed to compute_stats

prime_text(preamble: str | List[str], stimuli: str | List[str], image=None, separator: str | List[str] = ' ') → Tuple¶

Prepares a batch of input text into a format fit to run LM scoring on.

Parameters:

preamble (Union[str, List[str]]) – Batch of prefixes/prime/preambles on which the LM is conditioned.
stimuli (Union[str, List[str]]) – Batch of continuations that are scored based on the conditioned text (provided in the preamble). The positions of the elements match their counterparts in the preamble.

Returns:

Batch of formatted input that can be passed to compute_stats

next_word_distribution(queries: List, image: List, surprisal: bool = False)¶: Returns the log probability distribution of the next word.

compute_stats(batch: Iterable, rank: bool = False, prob: bool = False, base_two: bool = False, return_tensors: bool = False, bow_correction: bool = False) → Tuple[List[float], List[float]] | List[float]¶

Primary computational method that processes a batch of prepared sentences and returns per-token scores for each sentence. By default, returns log-probabilities.

Parameters:

batch (Iterable) – batched input as processed by prepare_text or prime_text.
rank (bool) – whether the model should also return ranks per word (based on the conditional log-probability of the word in context).
prob (bool) – whether the model should return probabilities instead of log-probabilities. Can only be True when base_two is False.
base_two (bool) – whether the base of the log should be 2 (usually preferred when reporting results in bits). Can only be True when prob is False.
return_tensors (bool) – whether the model should return scores as a list of tensors instead of a list of lists. This is important in some other convenient methods used in the package.
bow_correction (bool) – whether to apply the beginning of word correction, as pointed out in Pimentel and Meister (2024) and Oh and Schuler (2024).

Returns:

Either a tuple of lists, each containing probabilities and ranks per token in each sentence passed in the input.

Return type:

Union[Tuple[List[float], List[int]], List[float]]

sequence_score(text_batch, image_batch=None, reduction=<function VLMScorer.<lambda>>, base_two=False, bow_correction: bool = False)¶: TODO: reduction should be a string, if it’s a function, specify what kind of function. –> how to ensure it is always that type?

token_score(text_batch: str | List[str], image_batch=None, surprisal: bool = False, prob: bool = False, base_two: bool = False, rank: bool = False, decode: bool = True, bow_correction: bool = False, **kwargs) → List[Tuple[str, float]] | List[Tuple[str, float, int]]¶

For every input sentence, returns a list of tuples in the following format:: (token, score),

where score represents the log-probability (by default) of the token given context. Can also return ranks along with scores.

Parameters:

batch (Union[str, List[str]]) – a single sentence or a batch of sentences.
surprisal (bool) – If True, returns per-word surprisals instead of log-probabilities.
prob (bool) – If True, returns per-word probabilities instead of log-probabilities.
base_two (bool) – If True, uses log base 2 instead of natural-log (returns bits of values in case of surprisals)
rank (bool) – If True, also returns the rank of each word in context (based on the log-probability value)
bow_correction (bool) – whether to apply the beginning of word correction, as pointed out in Pimentel and Meister (2024) and Oh and Schuler (2024).

Returns:

A List containing a Tuple consisting of the word, its associated score, and optionally, its rank.

Return type:

Union[List[Tuple[str, float]], List[Tuple[str, float, int]]]

conditional_score(prefix: str | ~typing.List[str], stimuli: str | ~typing.List[str], image=None, separator: str = ' ', reduction: ~typing.Callable = <function VLMScorer.<lambda>>, prob: bool = False, base_two: bool = False, bow_correction: bool = False, **kw) → List[float]¶

Pooled estimates of sequence log probabilities (or some modification of it), given a prefix. Pooling is usually done using a function that is passed to the method.

Parameters:

prefix (Union[str, List[str]]) – a batch of prefixes or primes passed to the language model. This is what the sequence is conditioned on, and the model ignores the word probabilities of this part of the input in estimating the overall score.
stimuli (Union[str, List[str]]) – a batch of sequences (same length as prefix) that form the main input consisting of the sequence whose score you want to calculate.
reduction (Callable) – Reduction function, is selected to be lambda x: x.mean(0).item() by default, which stands for the avg. log-probability per token for each sequence in the batch.
bow_correction (bool) – whether to apply the beginning of word correction, as pointed out in Pimentel and Meister (2024) and Oh and Schuler (2024).
kw – model-specific keyword arguments to pass to the prepare_text function

Returns:

List of floats specifying the desired score for the stimuli part of the input, e.g., P(stimuli | preamble).

Return type:

List[float]