Annotation

Data Preparation

Preliminary Page Set Creation

This hard-coded script was used to create a PPA page set for our preliminary annotation efforts. Note: this script may not work.

This script takes three inputs:

  1. A directory for a PPA text corpus

  2. Our poetry testset file (CSV) that included manual identifications of pages containing poetry for several works in the PPA

  3. An output JSONL

The script then returned a CSV containing page-level metadata for every page of the PPA works covered in our poetry testet. This page-level metadata included an image path, if the page is known to contain poetry, and some work-level information.

Example usage:

python create_pageset.py ppa_corpus_dir poetry_testset.csv out.jsonl

Add Metadata

This script is used to prepare the page-level PPA corpus data for use in Prodigy annotation. It adds work-level metadata (title, author, year) in the locations that Prodigy requires for display, and allows adjusting image paths for display from the Prodigy interface. It assumes the input page corpus has corpus has already been annotated with image paths with an image_path attribute using corppa.utils.add_image_relpaths.

Example usage:

python add_metadata.py ppa_with_images.jsonl ppa_metadata.csv out.jsonl

Annotation Recipes

This module provides custom recipes for Prodigy annotation. These were created with page-level annotation in mind and assume a page is associated with both text and an image. Each recipe displays a page’s image and text side-by-side.

Recipes:

  • annotate_page_text: Annotate a page’s text.

  • annotate_text_and_image: Annotate both a page’s text and image side-by-side.

  • review_page_spans: Review existing page-level text annotations to produce a final, adjudicated set of annotations.

Referenced images must be served out independently for display; the image url prefix for images should be specified when initializing the recipe.

Example use:

prodigy annotate_page_text poetry_spans poetry_pages.jsonl --label POETRY,PROSODY -F annotation_recipes.py --image-prefix http://localhost:8000/

prodigy annotate_text_and_image poetry_text_image poetry_pages.jsonl -l POETRY -F annotation_recipes.py --image-prefix ../ppa-web-images -FM

prodigy review_page_spans adjudicate poetry_spans -l POETRY -F annotation_recipes.py --image-prefix ../ppa-web-images -FM --sessions alice,bob

Command Recipes

This module contains custom command recipes for Prodigy.

Recipes:

  • ppa-task-progress: Report the current progress for a PPA annotation task at the page and annotator level.

Example use:

prodigy ppa-stats task_id -F command_recipes.py

Process Adjudication Data

This script processes the adjudication data produced by Prodigy for our poetry detection task into two outputs:

  1. A JSONL file that compiles the annotation data into page-level records. So, each record contains some page-level metdata and the compiled list of poetry excerpts (if any) determined in the adjudication process.

  2. A CSV file containing excerpt-level data per line.

Note that the first file explicitly include information on the pages where no poetry was identified, while the second will only implicitly through absence and requires external knowledge of what pages were covered in the annotation rounds. So, the former is particularly useful for the evaluation process while the latter is better suited for building a final excerpt dataset.

Example command line usage:

python process_adjudication_data.py prodigy_data.jsonl adj_pages.jsonl adj_excerpts.csv