Utils

Filter Utility

Utility for filtering PPA full-text corpus to work with a subset of pages.

Currently supports the following types of filtering:

• List of PPA work ids (as a text file, id-per-line)

• CSV file specifying work pages (by digital page number) (csv, page-per-line)

• Filtering by key-value pair for either inclusion or exclusion

These filtering options can be combined, generally as a logical AND. Pages filtered by work ids or page numbers will be further filtered by the key-value logic. In cases where both work- and page-level filtering occurs, works not specified in the page filtering are included in full. Works that are specified in both will be limited to the pages specified in page-level filtering.

Filter methods can be run via command-line or python code. Filtering takes a jsonl file (compressed or not) as input, and will produce a jsonl file (compressed or not) as output. The input and output filenames can use any extension supported by orjsonl, with or without compression (e.g. .jsonl, .jsonl.gz, .jsonl.bz2).

Example command line usages:

corppa-filter path/to/ppa_pages.jsonl output/ppa_subset_pages.jsonl --idfile my_ids.txt

corppa-filter path/to/ppa_pages.jsonl output/ppa_subset_pages.jsonl --pg-file pages.csv --include key=value

Path Utilities

General-purpose methods for working with paths, PPA identifiers, and directories

corppa.utils.path_utils.decode_htid(encoded_htid: str) → str

Return original HathiTrust volume identifier from encoded version:

[library id].[encoded volume id]

Specifically, the volume-portion of the id undergoes the following character replacements:

"+" --> ":", "=" --> "/", "," --> "."

corppa.utils.path_utils.encode_htid(htid: str) → str

Returns the “clean” version of a HathiTrust volume identifier with the form:

[library id].[volume id]

Specifically, the volume-portion of the id undergoes the following character replacements:

":" --> "+", "/" --> "=", "." --> ","

corppa.utils.path_utils.find_relative_paths(base_dir: Path, exts: Iterable[str], follow_symlinks: bool = True, group_by_dir: bool = False) → Iterator[Path] | Iterator[tuple[Path, list[Path]]]

This method finds files anywhere under the specified base directory that match any of the specified file extensions (case insensitive), and returns a generator of path objects with a path relative to the base directory. File extensions should include the leading period, i.e. [".jpg", ".tiff"] rather than ["jpg", "tiff"].

For example, given a base directory a/b/c/images, an extension list of .jpg, and files nested at different levels in the hierarchy a/b/c/images/alpha.jpg, a/b/c/images/d/beta.jpg:

a/b/c/images
  |-- alpha.jpg
  +-- d
      |-- beta.jpg

The result will include the two items: alpha.jpg and d/beta.jpg

When group_by_dir is True, resulting files will be returned grouped by the parent directory. The return result is a tuple of a single pathlib.Path object for the directory and a list of pathlib.Path objects for the files in that directory that match the specified extensions. Given a hierarchy like this:

images/vol-a/
  |-- alpha.jpg
  |-- beta.jpg

the method would return (vol-a, [alpha.jpg, beta.jpg]).

corppa.utils.path_utils.get_gale_image_name(vol_id: str, page_num: int) → str

Return the Gale/ECCO page image filename for the given volume id and page number, e.g. CB0127060085_00050.TIF. Gale image filenames use a zero-padded page number with a trailing 0.

corppa.utils.path_utils.get_image_relpath(work_id: str, page_num: int) → Path

Get the (relative) image path for specified PPA work page

corppa.utils.path_utils.get_page_number(pagefile: Path) → str

Extract and return the page number from the filename for page-level content (e.g., image or text). Returns the page number as a string with leading zeros. (Note: logic is currently specific to Gale/ECCO file naming conventions.)

corppa.utils.path_utils.get_ppa_source(vol_id: str) → str

For a given volume id, return the corresponding source. Assume: * Gale volume ids begin with "CW0" or "CB0" * EEBO-TCP volume ids begin with A * Hathitrust volume ids contain a "."

corppa.utils.path_utils.get_stub_dir(source: str, vol_id: str) → Path

Returns the stub directory path (pathlib.Path) for the specified volume (vol_id)

For Gale, the path is formed from every third number (excluding the leading 0) of the volume identifier.

Ex. CB0127060085 --> 100

For HathiTrust, we use the Stubbytree directory specification created by HTRC. The path is composed of two directories: (1) the library portion of the volume identifier and (2) every third character of the encoded volume identifier.

Ex. mdp.39015003633594 --> mdp/31039

corppa.utils.path_utils.get_vol_dir(vol_id: str) → Path

Returns the volume directory (pathlib.Path) for the specified volume (vol_id)

corppa.utils.path_utils.get_volume_id(work_id: str) → str

Extract volume id from PPA work id

• For full works, volume ids and work ids are the same.

• For excerpts, the work id is composed of the prefix followed by “-p” and the starting page of the excerpt.

Generate PPA Page Set

Utility for generating a PPA page set.

This method takes three inputs: (1) an input csv, (2) an output csv, and (3) the size of the page set.

The input CSV file must have the following fields:

• work_id: PPA work id

• page_start: Starting index for page range being considered for this work

• page_end: Ending index for page range being considered for this work

• poery_pages: Comma separated list of page numbers containing poetry

The pages are selected as follows:

• First, all pages with poetry are selected

• Then, all remaining pages are chosen randomly (proportionately by work)

The resulting output CSV file has the following fields:

• work_id: PPA work id

• page_num: Digital page number

Example usage:

python generate_page_set.py input.csv output.csv 300

Add Image (Relative) Paths

This script adds image relative paths to the PPA full-text corpus. Optionally, a file extension (e.g., .jpg) can be provided to be used for all relative image paths instead of defaulting to their source-level defaults.

Example usage:

python add_image_relpaths.py ppa_corpus.jsonl ppa_with_images.jsonl

python add_image_relpaths.py ppa_corpus.jsonl ppa_with_images.jsonl --ext=.jpg

Build Text Corpus

Script for building a text corpus file (JSONL) from a directory of texts.

This script converts each text (.txt) file within an input directory (including nested files), and compiles them into a single output text corpus where each record corresponds to a single text file with the following fields:

• id: The name of the file (without prefix)

• text: The text of the file (assumes UTF-8 formatting)

Note that the output file can also be written in any compressed form supported by orjsonl. If no suffix is provided, .jsonl will be used.

Example usage:

python build_text_corpus.py input_dir out_corpus

python build_text_corpus.py input_dir out_corpus.jsonl

python build_text_corpus.py input_dir out_corpus.jsonl.gz

Dataset Preparation

This is a utility scripto help prep the PPA metadata/text/image dataset for publication.

It should be used to align page images with text to create a page_image.tar file and update the ppa_pages.jsonl file with image paths and optionally old text, when OCR has changed.

Handles the following formats:

• HathiTrust dataset zip files

• HathiTrust manually downloaded zip files (full or excerpted) for 1930s content

• Gale/ECCO image directory, using existing corppa path utils and naming conventions

For HathiTrust dataset zip files, the script checks for page alignment; if average text similarity is below our trusted threshold, does a best-effort alignment based on high-confidence, unambiguous page matches which are used to infer matches for neighboring pages.

For convenience, a slurm sbatch file is provided in scripts/dataset_prep.sbatch

Supports interruption and continuing partial work, so that alignment can be completed in multiple passes.

corppa.utils.dataset_prep.add_image_paths_to_metadata(corpus_dir: Path, output_dir: Path) → None

Update metadata with image_file and image_path fields for works with images.

corppa.utils.dataset_prep.add_zip_file_to_tar(zipfile: ZipFile, zip_filename: str, tar: TarFile, tar_file_path: str) → None

Add a single file from an open ZipFile to an open TarFile without extracting to disk. tar_file_path sets the path within the tar archive.

corppa.utils.dataset_prep.align_pages(work_id: str, pages_df: DataFrame, zipfile: ZipFile) → dict

Determine alignment between pages in different versions of a HathiTrust work. Returns a dict mapping each page id to a (page_filename, new_ocr) tuple, where new_ocr is the zip page text when it differs from the original page text and None otherwise. Returns an empty dict if no alignment could be determined.

corppa.utils.dataset_prep.align_shifted_pages(pages_df: DataFrame, zip_pages_df: DataFrame, detailed: bool = False) → DataFrame

Align corpus pages to zip page filenames when page order has shifted between versions. Shifts are determined by matching pages with sufficient text in the first dataframe to pages in the zip file using normalized indel similarity (rapidfuzz.fuzz.ratio). Short pages and pages with low-confidence matches are aligned based on the shift of the nearest preceding alignment, or nearest following alignment if no preceding alignment.

Returns a DataFrame with id, page_filename, and new_ocr columns where each row is the determined alignment; returns an empty dataframe if alignment could not be determined.

When detailed is True, the returned frame instead includes the per-page diagnostic columns useful for reviewing/visualizing an alignment: id, order (original page order), aligned_order (mapped zip order), page_filename, cdist_best_score (best rapidfuzz.cdist score, 0-100, for every scored/long page; null for short pages), shift (trusted anchor shift, null for filled pages), inferred_shift (shift actually applied), and the raw text / zip_text for each page. Derived review fields (is_anchor/is_matched flags, text lengths, snippets, extra scoring) are added by review_alignment rather than here.

corppa.utils.dataset_prep.find_corpus_file(corpus_dir: Path, filenames: list[str]) → Path

Return the first existing file in corpus_dir from filenames (checked in order). Raises FileNotFoundError if none exist.

corppa.utils.dataset_prep.get_ht1930_work_ids(metadata_path: Path) → dict[str, str | None]

Load work-level metadata and return a dictionary of work_id values for HathiTrust volumes published in 1930, with their corresponding digital page range (excerpts only), OR any other volumes that were added to PPA after the text/image dataset was requested.

These volumes have zip files with different naming convention and structure and must be handled differently.

corppa.utils.dataset_prep.get_ht_zipfile_path(work_id: str, image_dir: Path) → Path

Return the expected HathiTrust zip path for a work under image_dir. The zip is named from the (encoded) HathiTrust id without institution prefix.

corppa.utils.dataset_prep.get_zip_image_names(zipfile: ZipFile) → dict[str, str]

HathiTrust zip files include images in multiple formats, sometimes mixed even within a single zipfile. Use the ZipFile namelist to generate and return a dictionary of all files with supported image extensions. Dictionary key is the file basename, value is the full path within the zip.

corppa.utils.dataset_prep.get_zip_textfiles(zipfile: ZipFile) → Iterator[tuple[str, str]]

Return a generator of text files from an open zip archive. Returns tuples of (file_id, content) where file_id is the stem of the filename.

corppa.utils.dataset_prep.get_zipfile_pages(zipfile) → DataFrame

Load text files from a HathiTrust zipfile into a polars dataframe.

Returned dataframe has the following columns:

• page_filename

• text: full text contents of the page

• page_id: numeric portion extracted from page_filename

• order: numeric version of page_id

• text_len: number of characters in text

corppa.utils.dataset_prep.longest_increasing_subseq(values: ndarray) → ndarray

Return the indices (into values) of the longest strictly increasing subsequence. When several subsequences tie for longest, one is returned deterministically (same input always yields the same result); which of the tied subsequences is chosen is not part of the contract, since downstream alignment depends only on the derived shifts and the rapidfuzz-based dedup, not on the specific anchor set.

Used by align_shifted_pages to filter high-confidence mappings down to a set of “anchor” matches that enforce sequential alignment while allowing for gaps on either side.

We use this to reduce a set of candidate page matches down to a monotonic, conflict-free set of “anchors”. Each confident match pairs an original page with a zip page (identified by its order, the digital page sequence). Reading those zip page orders in original-page order, a valid alignment must be strictly increasing: later original pages can only map to later zip pages, and each zip page may be claimed at most once (strict increase => no repeats). Any confident match that would break that ordering - e.g. repeated boilerplate pages that all matched the same zip page, or a match that jumps backwards - is dropped by keeping only the longest increasing run.

This is the classic O(n log n) patience-sorting algorithm.

corppa.utils.dataset_prep.main()

Parse command-line arguments and prepare a full-text dataset.

corppa.utils.dataset_prep.open_ht_zipfile(zipfile_path: Path | None) → Iterator[ZipFile | None]

Open a HathiTrust image zip file, yielding the open ZipFile. Yields None (without raising) when zipfile_path is None or does not exist, so callers can uniformly skip works whose zip is missing.

corppa.utils.dataset_prep.plot_alignment(review_df: DataFrame)

Visualize a review frame from review_alignment as an Altair chart.

Original pages are drawn on one row and their mapped zip pages on a second row, both positioned by page order along x. A line connects each original page to the zip page it maps to, so the horizontal offset of that line is the shift; parallel lines mean a consistent shift, and crossings/gaps stand out. Points are colored by alignment status (anchor / inferred / unmatched) and sized by match score. Hover shows a compact mapping/score line (e.g. “page 24 -> zip 12 | best 100 | aligned 100”) and a text snippet.

Expects the derived columns added by review_alignment (snippets, match_score). Requires altair (install the notebooks extra).

corppa.utils.dataset_prep.process_gale_work(work_id: str, pages: list[dict], image_dir: Path, tar: TarFile) → Iterator[dict]

Add available Gale page images to a work and yield its page records.

corppa.utils.dataset_prep.process_ht1930_work(work_id: str, pages: list[dict], image_dir: Path, tar: TarFile, digital_page_range: str | None = None) → Iterator[dict]

Images for HathiTrust works published in 1930 and imported manually in 2026 were downloaded manually from the HT web interface. The zip files only contain images (no text files, no nested directories) and were downloaded close to when the works were imported, so they do not require page re-alignment. Pass digital_page_range for excerpts to find the correct zipfile, which includes digital page range.

Find and load the zip file, and map images to pages based on filename.

corppa.utils.dataset_prep.process_ht_work(work_id: str, pages: list[dict], image_dir: Path, tar: TarFile) → Iterator[dict]

Align HathiTrust pages with images and yield the resulting page records.

corppa.utils.dataset_prep.process_work(work_id: str, pages: list[dict], image_dir: Path, tar: TarFile | None, ht1930_work_ids: dict[str, str | None] | None = None) → Iterator[dict]

Process one work using the source-specific image handling strategy.

corppa.utils.dataset_prep.review_alignment(work_id: str, pages: DataFrame | list[dict], zipfile_path: Path | str) → DataFrame

Run the shifted-page alignment for a single work and return a per-page frame for review/visualization. Intended for notebook use.

Builds on align_shifted_pages(detailed=True) and adds derived review columns: is_anchor / is_matched (flags inferred from shift / page_filename), text_len / zip_text_len (character counts), text_snippet / zip_text_snippet (first-line hover snippets), and match_score (the rapidfuzz.fuzz.ratio, 0-100, between each matched page and its actual aligned zip page) so it can be compared against cdist_best_score (the best score against any zip page).

pages may be a page DataFrame (with id, order/text columns) or a list of page dicts; zipfile_path is a path to the work’s zip file.

corppa.utils.dataset_prep.zip_image_filenames(zipfile: ZipFile) → dict[int, str]

Map numeric digital page sequence number to image filename within a HathiTrust zip.

Used for image-only volume zip files. Images are TIF or JPG only; base filename is numeric with leading zeros. Files with a non-numeric stem (e.g. stray cover/metadata images) are skipped.

Dataset Metadata Refinement

Utility script for PPA full-text dataset publication prep.

Takes a manually cleaned version of PPA metadata in CSV format, and propagates refined author and pub_place fields to CSV and JSON metadata in the specified ppa_corpus folder.

corppa.utils.dataset_refine.main()

Parse command-line arguments and refine a PPA corpus metadata directory.

corppa.utils.dataset_refine.refine_metadata(refined_metadata: Path, csv_metadata_path: Path, json_metadata_path: Path) → None

Propagate refined author and publication-place values to corpus metadata.