Utils

Filter Utility

Utility for filtering PPA full-text corpus to work with a subset of pages.

Currently supports the following types of filtering:

  • List of PPA work ids (as a text file, id-per-line)

  • CSV file specifying work pages (by digital page number) (csv, page-per-line)

  • Filtering by key-value pair for either inclusion or exclusion

These filtering options can be combined, generally as a logical AND. Pages filtered by work ids or page numbers will be further filtered by the key-value logic. In cases where both work- and page-level filtering occurs, works not specified in the page filtering are included in full. Works that are specified in both will be limited to the pages specified in page-level filtering.

Filter methods can be run via command-line or python code. Filtering takes a jsonl file (compressed or not) as input, and will produce a jsonl file (compressed or not) as output. The input and output filenames can use any extension supported by orjsonl, with or without compression (e.g. .jsonl, .jsonl.gz, .jsonl.bz2).

Example command line usages:

corppa-filter path/to/ppa_pages.jsonl output/ppa_subset_pages.jsonl --idfile my_ids.txt

corppa-filter path/to/ppa_pages.jsonl output/ppa_subset_pages.jsonl --pg-file pages.csv --include key=value

Path Utilities

General-purpose methods for working with paths, PPA identifiers, and directories

corppa.utils.path_utils.decode_htid(encoded_htid: str) → str

Return original HathiTrust volume identifier from encoded version:

[library id].[encoded volume id]

Specifically, the volume-portion of the id undergoes the following character replacements:

"+" --> ":", "=" --> "/", "," --> "."
corppa.utils.path_utils.encode_htid(htid: str) → str

Returns the “clean” version of a HathiTrust volume identifier with the form:

[library id].[volume id]

Specifically, the volume-portion of the id undergoes the following character replacements:

":" --> "+", "/" --> "=", "." --> ","
corppa.utils.path_utils.find_relative_paths(base_dir: Path, exts: Iterable[str], follow_symlinks: bool = True, group_by_dir: bool = False) → Iterator[Path] | Iterator[tuple[Path, list[Path]]]

This method finds files anywhere under the specified base directory that match any of the specified file extensions (case insensitive), and returns a generator of path objects with a path relative to the base directory. File extensions should include the leading period, i.e. [".jpg", ".tiff"] rather than ["jpg", "tiff"].

For example, given a base directory a/b/c/images, an extension list of .jpg, and files nested at different levels in the hierarchy a/b/c/images/alpha.jpg, a/b/c/images/d/beta.jpg:

a/b/c/images
  |-- alpha.jpg
  +-- d
      |-- beta.jpg

The result will include the two items: alpha.jpg and d/beta.jpg

When group_by_dir is True, resulting files will be returned grouped by the parent directory. The return result is a tuple of a single pathlib.Path object for the directory and a list of pathlib.Path objects for the files in that directory that match the specified extensions. Given a hierarchy like this:

images/vol-a/
  |-- alpha.jpg
  |-- beta.jpg

the method would return (vol-a, [alpha.jpg, beta.jpg]).

corppa.utils.path_utils.get_image_relpath(work_id: str, page_num: int) → Path

Get the (relative) image path for specified PPA work page

corppa.utils.path_utils.get_page_number(pagefile: Path) → str

Extract and return the page number from the filename for page-level content (e.g., image or text). Returns the page number as a string with leading zeros. (Note: logic is currently specific to Gale/ECCO file naming conventions.)

corppa.utils.path_utils.get_ppa_source(vol_id: str) → str

For a given volume id, return the corresponding source. Assume: * Gale volume ids begin with "CW0" or "CB0" * EEBO-TCP volume ids begin with A * Hathitrust volume ids contain a "."

corppa.utils.path_utils.get_stub_dir(source: str, vol_id: str) → Path

Returns the stub directory path (pathlib.Path) for the specified volume (vol_id)

For Gale, the path is formed from every third number (excluding the leading 0) of the volume identifier.

Ex. CB0127060085 --> 100

For HathiTrust, we use the Stubbytree directory specification created by HTRC. The path is composed of two directories: (1) the library portion of the volume identifier and (2) every third character of the encoded volume identifier.

Ex. mdp.39015003633594 --> mdp/31039
corppa.utils.path_utils.get_vol_dir(vol_id: str) → Path

Returns the volume directory (pathlib.Path) for the specified volume (vol_id)

corppa.utils.path_utils.get_volume_id(work_id: str) → str

Extract volume id from PPA work id

  • For full works, volume ids and work ids are the same.

  • For excerpts, the work id is composed of the prefix followed by “-p” and the starting page of the excerpt.

Generate PPA Page Set

Utility for generating a PPA page set.

This method takes three inputs: (1) an input csv, (2) an output csv, and (3) the size of the page set.

The input CSV file must have the following fields:

  • work_id: PPA work id

  • page_start: Starting index for page range being considered for this work

  • page_end: Ending index for page range being considered for this work

  • poery_pages: Comma separated list of page numbers containing poetry

The pages are selected as follows:

  • First, all pages with poetry are selected

  • Then, all remaining pages are chosen randomly (proportionately by work)

The resulting output CSV file has the following fields:

  • work_id: PPA work id

  • page_num: Digital page number

Example usage:

python generate_page_set.py input.csv output.csv 300

Add Image (Relative) Paths

This script adds image relative paths to the PPA full-text corpus. Optionally, a file extension (e.g., .jpg) can be provided to be used for all relative image paths instead of defaulting to their source-level defaults.

Example usage:

python add_image_relpaths.py ppa_corpus.jsonl ppa_with_images.jsonl

python add_image_relpaths.py ppa_corpus.jsonl ppa_with_images.jsonl --ext=.jpg

Build Text Corpus

Script for building a text corpus file (JSONL) from a directory of texts.

This script converts each text (.txt) file within an input directory (including nested files), and compiles them into a single output text corpus where each record corresponds to a single text file with the following fields:

  • id: The name of the file (without prefix)

  • text: The text of the file (assumes UTF-8 formatting)

Note that the output file can also be written in any compressed form supported by orjsonl. If no suffix is provided, .jsonl will be used.

Example usage:

python build_text_corpus.py input_dir out_corpus

python build_text_corpus.py input_dir out_corpus.jsonl

python build_text_corpus.py input_dir out_corpus.jsonl.gz