Component relationships
Collection and restart model
An object-oriented collection service composes source adapters, an HTTP broker, extraction workers and SQLite persistence. Adapters discover and retrieve source-family inputs; the service owns scheduling, pagination and processing.
Durable item states, page cursors and heartbeat leases support restart and reclamation of interrupted work. Shared per-origin request handling and bounded extraction concurrency coordinate parallel sources. GUI jobs expose batch progress and stop requests; command-line watch mode runs due-source collection.
Extraction, validation and provenance
The collector deduplicates inputs, retrieves text/PDFs, applies OCR where required and asks a configurable model endpoint for structured observations. Local validators check JSON shape, taxonomy, dates, quantities and supporting evidence.
Validation is per observation: a nonconforming proposal can be rejected without discarding valid peers. Document/configuration fingerprints identify what was processed; stored observations retain the source evidence when reconciled into events.
Review and separate output paths
The control room presents searchable observations, related event evidence and append-only review/correction history. A correction preserves the original extraction; marking the corrected observation supported is a separate decision.
Periodic indicators use a distinct artifact preparation, processing and persistence path. Normalized observation and event-summary exports are available through the CLI; the GUI does not provide a dedicated indicator-series or export workspace.
Intended modelling use
Public Signals prepares normalized external data intended to improve demand and lead-time forecasts. The method of incorporating these signals into the forecasting models is not yet determined. Collection-health counts and extraction confidence are not evidence of forecasting improvement or a business-risk score.
Implementation notes
Adapter composition
Adapters implement discovery and retrieval for a source family. The collection service controls run ownership, pagination, fetching and extraction; adapters do not each operate an independent scheduler.
Extraction is a proposal
An API model proposes JSON observations. Local schema, taxonomy, date, number and evidence-grounding checks determine whether those observations are accepted.
Restart semantics
Durable item states and a heartbeat lease record which collector owns a run. A later process can reclaim interrupted work rather than duplicating an untracked scrape.