Poetry Detection

Core objects

Custom data type for poetry excerpts identified with the text of PPA pages.

corppa.poetry_detection.core.DETECTION_METHODS = {'adjudication': 'a', 'manual': 'm', 'passim': 'p', 'xml': 'x'}

Supported detection methods with corresponding prefixes

class corppa.poetry_detection.core.Excerpt(*, page_id: str, ppa_span_start: int, ppa_span_end: int, ppa_span_text: str, detection_methods: set[str], notes: str | None = None)

A detected excerpt of poetry within a PPA page text. Excerpt objects are immutable.

correct_page_excerpt(page_text: str) → Self

For an excerpt that may have undergone textual transformations during the detection process, this method attemps to correct the excerpt such that the returned excerpt has the indicies and text correpsonding to the original page text (rather than the transformed version). The correspondence is determined using the Needleman-Wunsch algorithm.

Warning: If the wrong page text is passed in, this method will still find an alignment, but “correct” the excerpt… but not to something meaningful. In the future, we may add support to guard against this issue.

detection_methods: set[str]

Detection methods

excerpt_id: str

excerpt identifier

classmethod field_types() → dict[str, Any]

Return a dictionary of field names and corresponding types for this class.

classmethod fieldnames(required_only=False) → list[str]

Return a list of names for the fields in this class, in order. Takes an optional parameter required_only to return the list of fields that are required for initialization.

classmethod from_dict(d: dict) → Excerpt

Constructs a new Excerpt from a dictionary in the format generated by meth:Excerpt.to_dict or meth:Excerpt.to_csv. Input values for set fields are converted with input_to_set(); input values for integer fields support conversion from string.

notes: str | None = None

Optional notes

page_id: str

page id

ppa_span_end: int

ppa span end index

ppa_span_start: int

ppa span start index

ppa_span_text: str

ppa span text

strip_whitespace() → Self

Return a copy of this excerpt with any leading and trailing whitespace removed from the text and start and end indices updated to match any changes.

to_csv() → dict[str, int | str]

Returns a CSV-friendly dict of the poem excerpt. Note that like to_dict unset fields are not included.

to_dict() → dict[str, Any]

Returns a JSON-friendly dict of the poem excerpt. Note that unset optional fields are not included.

class corppa.poetry_detection.core.LabeledExcerpt(*, page_id: str, ppa_span_start: int, ppa_span_end: int, ppa_span_text: str, detection_methods: set[str], notes: str | None = None, poem_id: str, ref_corpus: str, ref_span_start: int | None = None, ref_span_end: int | None = None, ref_span_text: str | None = None, alt_poem_ids: set[str] | None = None, identification_methods: set[str])

An identified excerpt of poetry within a PPA page text.

alt_poem_ids: set[str] | None = None

set of alternate poem ids, for merged excerpts with multiple ids

classmethod from_excerpt(ex: Excerpt, **kwargs: dict) → LabeledExcerpt

Create a LabeledExcerpt using an Excerpt as a starting point and supplying data for additional fields.

identification_methods: set[str]

Identification methods

poem_id: str

poem id

ref_corpus: str

reference corpus id

ref_span_end: int | None = None

reference span end index

ref_span_start: int | None = None

reference span start index

ref_span_text: str | None = None

reference span text

corppa.poetry_detection.core.MULTIVAL_DELIMITER = '; '

character to use when converting sets to and from delimited string

class corppa.poetry_detection.core.Span(start: int, end: int, label: str)

Span object representing a Pythonic “closed open” interval

end: int

end index

has_overlap(other: Span, ignore_label: bool = False) → bool

Returns whether this span overlaps with the other span. Optionally, span labels can be ignored.

is_exact_match(other: Span, ignore_label: bool = False) → bool

Checks if the other span is an exact match. Optionally, ignores span labels.

label: str

label for the span

overlap_factor(other: Span, ignore_label: bool = False) → float

Returns the overlap factor with the other span. Optionally, span labels can be ignored for this calculation.

The overlap factor is defined as follows:

• If no overlap (overlap = 0), then overlap_factor = 0.

• Otherwise, overlap_factor = overlap_length / longer_span_length

So, the overlap factor has a range between 0 and 1 with higher values corresponding to a higher degree of overlap.

overlap_length(other: Span, ignore_label: bool = False) → int

Returns the length of overlap between this span and the other span. Optionally, span labels can be ignored for this calculation.

start: int

start index

corppa.poetry_detection.core.field_real_type(field_type) → type

Return the real type for a dataclass field type annotation. For unions or optional values (e.g. Optional[int]), returns the first non-None type; for type aliases (e.g. set[str]), returns the original type that was used to create the alias. For example:

• int -> int

• Optional[int] -> int

• set[str] -> set

corppa.poetry_detection.core.input_to_set(input_val: list | str | set) → set

Convert supported inputs to set; intended for convenience when initializing Excerpt.detection_methods and LabeledExcerpt.identification_methods.

Reference Corpora

class corppa.poetry_detection.ref_corpora.BaseReferenceCorpus

Base class for reference poetry corpora, with corpus identifier and methods to access metadata and text content.

static calculate_poem_length(text: str) → dict[str, int]

Calculate poem length metrics from text content. Takes the text of the poem and returns a dictionary num_lines (non-blank lines), num_words, and char_len.

get_metadata_df(poem_length=False) → DataFrame

Minimal common poetry metadata for use across reference corpora. Should return a pl.DataFrame with poem_id, author, title, and ref_corpus for each poem in this corpus. Optionally, should return information about poem length (number of characters and lines in the text).

get_text_corpus() → Generator[dict[str, str]]

Minimal text record for reference corpora. Should yield a dictionary with id and text for each poem in this corpus.

class corppa.poetry_detection.ref_corpora.ChadwyckHealey(config_opts: CorpusConfig)

Reference corpus based on a filtered subset of Chadwyck-Healey poetry collection. Requires a directory of plain text files and a metadata csv file. Uses Chadwyck-Healey identifiers for poem_id.

corpus_id: str = 'chadwyck-healey'

id for this reference corpus: chadwyck-healey

get_metadata_df(poem_length=False) → DataFrame

Minimal common poetry metadata for use across reference corpora. Should return a pl.DataFrame with poem_id, author, title, and ref_corpus for each poem in this corpus. Optionally, should return information about poem length (number of characters and lines in the text).

class corppa.poetry_detection.ref_corpora.InternetPoems(config_opts: CorpusConfig)

Curated corpus of poems with plain text content sourced from the internet, for high priority sources known to occur in excerpts, including full text of Shakespeare’s plays. Metadata was originally based on filename (naming convention of Firstname-Lastname_Poem-Title.txt), but has since been converted to a CSV file for correction and augmentation. The text filename without extension is used as the poem_id.

corpus_id: str = 'internet_poems'

id for this reference corpus: internet_poems

get_metadata_df(poem_length=False) → DataFrame

Minimal common poetry metadata for use across reference corpora. Should return a pl.DataFrame with poem_id, author, title, and ref_corpus for each poem in this corpus. Optionally, should return information about poem length (number of characters and lines in the text).

class corppa.poetry_detection.ref_corpora.LocalTextCorpus(config_opts: CorpusConfig)

Base class for reference corpus where text content is provided as a set of text files in a directory or tar.gz. On initialization, configures data path based on configured base dir and corpus default or any overrides, and validates that the path exists and is a directory. Provides get_text_corpus() for generating text corpus from the file system.

get_text_corpus(disable_progress: bool = True) → Generator[dict[str, str]]

Minimal text record for reference corpora. Should yield a dictionary with id and text for each poem in this corpus.

corppa.poetry_detection.ref_corpora.METADATA_SCHEMA = {'author': String, 'char_len': Int64, 'num_lines': Int64, 'num_words': Int64, 'poem_id': String, 'ref_corpus': String, 'title': String}

schema for reference corpora metadata pl.DataFrame

class corppa.poetry_detection.ref_corpora.OtherPoems(config_opts: CorpusConfig)

A metadata-only reference corpus with metadata for poems that have been identified but for which we do not have full text. Poem identifiers are constructed from author and title using the same convention as InternetPoems.

Does not provide an implementation for get_text_corpus().

config: CorpusConfig

config with metadata_path for URL or local path to metadata

corpus_id: str = 'other'

id for this reference corpus (currently “other”)

get_metadata_df(poem_length=False) → DataFrame

Minimal common poetry metadata for use across reference corpora. Should return a pl.DataFrame with poem_id, author, title, and ref_corpus for each poem in this corpus. Optionally, should return information about poem length (number of characters and lines in the text).

corppa.poetry_detection.ref_corpora.all_corpora() → list[BaseReferenceCorpus]

Convenience access to all reference corpora, for generating compiled versions of reference data.

corppa.poetry_detection.ref_corpora.compile_metadata_df(poem_length=False) → DataFrame

Compile poetry metadata from all reference corpora into a single polars DataFrame with reference corpus ids.

corppa.poetry_detection.ref_corpora.fulltext_corpora() → list[BaseReferenceCorpus]

Convenience access to all full-text reference corpora, for generating compiled metadata and text.

corppa.poetry_detection.ref_corpora.save_poem_metadata(output_file: Path, excerpts_df: DataFrame | None = None, poem_clusters_df: DataFrame | None = None)

Generate and save compiled poetry metadata as a data file in the poem dataset. Loads and compiles metadata for all reference corpora, including poem length calculations (compile_metadata_df()) and saves the result to the specified output_file. When the optional excerpts_df is present, calculates work-level excerpt total for poems based on primary poem id (number of excerpts, number of PPA works, number of PPA pages). When the optional poem_clusters_df is provided, adds a cluster_id field to poems known to be duplicates, near-duplicates or subsets.

Scripts

refmatcha

🎶🍵 matcha matcha poem / This script is gon / na find your poems / matcha matcha poem 🎶🍵

refmatcha identifies poem excerpts by matching against a local collection of reference poems. It takes in a CSV of unidentified excerpts and outputs a CSV of labeled excerpts for those excerpts it is able to identify. By default, the output file is created in the same directory as the input with the same name plus _matches; i.e., given an input file round1_excerpts.csv, refmatcha will output identified excerpts to round1_excerpts_matches.csv. To override this, specify an output filename with –output or -o.

Setup:

Download and extract poetry-ref-data.tar.bz2 from /tigerdata/cdh/prosody/poetry-detection. You should extract it in the same directory where you plan to run this script. The script will compile reference content into full-text and metadata parquet files on the first run; to force recompilation, rename or remove the parquet files.

Example usage:

refmatcha round1_excerpts.csv
refmatcha round1_excerpts.csv --output round1_matches.csv

Merge excerpts

This script and associated method merges labeled and unlabeled poem excerpts with matching spans in the PPA page text.

It takes two or more input files of excerpt data (labeled or unlabeled) in CSV format, merges any excerpts that can be combined, and outputs a CSV with the updated excerpt data. All excerpts in the input data files are preserved in the output, whether they were merged with any other records or not. This means that in most cases, the output will likely be a mix of labeled and unlabeled excerpts.

Merging logic is as follows:

• Excerpts are grouped based on exact span match in PPA text (i.e., the combination of page_id, ppa_span_start, and ppa_span_end) even when poem identifications differ, and combined as follows:

  • Excerpts are sorted by poem_id, ref_span_start, and passim match length with nulls last and longest passim match first. Reference information (poem_id, ref_span_start, ref_span_end, ref_span_text, ref_corpus) is taken from the first excerpt in the group.

  • When merged excerpts have different poem identifications, all unique poem ids after the first are collected into alt_poem_ids

  • The detection_methods and identification_methods fields are combined to the unique set of methods used in the merged excerpts.

  • The notes field is combined with the set of all unique content from notes in merged excerpts with an additional note about the merge.

Example usage:

merge-excerpts adjudication_excerpts.csv labeled_excerpts.csv -o merged_excerpts.csv

Limitations:

• Merge logic collapses different poem ids that may not correspond; they may be subsets of the same poem, different editions, or entirely different poems. Alternate poem ids are preserved in alt_poem_ids.

• Currently supports CSV input and output only.

corppa.poetry_detection.merge_excerpts.identify_overlapping_excerpts(excerpts_df: DataFrame, min_overlap_factor: float = 0.98, min_overlap_chars: int = 10) → DataFrame

Takes a DataFrame of excerpts and identifies pairs of overlapping excerpts. Overlapping excerpts are on the same page, with some shared span of text. We exclude short overlaps based on the minimum character parameter, and an overlap factor, which is calculated by the length of the shared span divided by the length of the longer of the two spans. Note that this will typically not return small excerpts completely inside another larger span.

Returns a DataFrame of excerpt pairs with columns for page id, pairs of excerpt ids, overlap length, and overlap factor.

corppa.poetry_detection.merge_excerpts.merge_excerpt_groups(grouped_df: GroupBy, merge_reason: str = 'ppa exact span') → DataFrame

Takes a GroupBy dataframe of excerpts (created by calling group_by), and combines groups of excerpts into merged excerpts. Fields are expected to correspond to labeled excerpts (LabeledExcerpt), and the dataframe should be pre-sorted so the preferred excerpt comes first, since in several cases the first value is the one preserved in the merge. Merge logic is as follows:

• first ppa_span_text, poem_id, reference corpus values (ref_corpus, ref_span_start, ref_span_end, ref_span_text)

• combined unique set of detection methods and identification methods

• combined unique set of notes

• updated excerpt id

• any additional poem ids are listed in alt_poem_ids

After merging, the notes field is updated with text documenting the merge with the specified reason (by default, exact span in PPA), and the number of excerpts that were merged.

corppa.poetry_detection.merge_excerpts.merge_excerpts(df: DataFrame, disable_progress=True, verbose=False) → DataFrame

Takes a polars DataFrame that includes labeled or unlabeled excerpts (fields correspond to LabeledExcerpt), and merges excerpts based on page_id and ppa span (ppa_span_start and ppa_span_end). For now, merging is only done on the simple cases where PPA excerpt text bounds match exactly. The best match is prioritized, based on passim match length; alternate poem ids are preserved in alt_poem_ids.

When excerpts are merged, the detection_methods, identification_methods, and notes fields are all combined to preserve information.

Returns a dataframe that contains all unique excerpts and merged versions of duplicate excerpts.