Prodigy v1.13.1: compare spaCy and LLM annotations

Uploaded on August 29, 2023, this tutorial covers Prodigy v1.13.1's models-as-annotators workflow. Vincent, a Prodigy developer at Explosion, compares a local spaCy English core web medium model with a spaCy-LLM pipeline using a GPT 3.5 endpoint. The setup reflects that release; the demonstrated LLM calls go to a vendor over the network.

The task is named entity recognition on news headlines, limited to person and organization labels. Vincent first inspects the statistical model's predictions, then stores them in Prodigy's database with a model annotation recipe and an annotator name. For the second model, he configures label definitions and prompt examples, assembles the pipeline on disk, and supplies its component name and endpoint environment variables.

Prodigy's review interface puts the two sets of predictions beneath an editable final annotation. Examples include an overextended person name, "Bill Maher concert," and an organization the statistical model missed, Alibaba. An auto-accept option skips agreements so the reviewer can focus on disagreements.

Vincent warns that agreement can still mean both models are wrong, so auto-accepting may introduce bad training data. He also notes vendor request costs and slower annotation, and suggests local pattern-matching pipelines as another option. He presents disagreement as a way to prioritize human review, not proof that an annotation is correct.