Verified research pipeline
A research batch that checks itself before the client sees it. Sixty rows in, fifty-two delivered, eight held back with the reason attached.
Research and list-building jobs fail in a quiet way: the file looks complete. Missing sources, dead links, placeholder emails and duplicate rows all survive an eyeball check and only surface after the campaign is sent. This build puts the checking in code, before delivery, and sends anything unverifiable to a person instead of padding the batch.
The brief
A repeat research brief arrives through a form: topic, batch size, minimum source quality. The pipeline creates a batch record, sources the rows, and runs every row through a deterministic gate. Rows that pass are filed to the delivered list. Rows that fail are written to an exception queue with defect codes, and a pending decision row is created for each one. Nothing is silently dropped and nothing is silently accepted.
Evidence from the demo run
Batch RB-470309, topic "Industrial suppliers in Ontario", minimum source quality "company website, dated in the last 90 days". Arithmetic closes: 52 verified plus 8 exceptions equals 60. All 52 clean rows passed; all 8 seeded defects were caught.
The rules, in code
| Defect | Rule that catches it | Seeded in the run |
|---|---|---|
| missing_company | Every required field must be non-empty | 1 row |
| missing_source_url | Each row must carry the URL it came from | 1 row |
| dead_source | Source host must not be a dead or placeholder domain | 1 row |
| placeholder_email | Email domain must not be example, test or invalid | 1 row |
| duplicate_id | Stable ID must be unique inside the batch | 1 row |
| name_mismatch | Contact name must be a plausible full name | 1 row |
| bad_category | Category must be one of the brief's allowed values | 1 row |
| stale_source | Source date must be inside the freshness window | 1 row |
The gate runs on the batch, not one row at a time, so duplicates are caught against the whole batch. Each pass writes a stable row ID, the source URL and the checked timestamp into the audit tab, so a later reviewer can re-check any row without trusting the file.
When it breaks
The workflow has a failure alert wired to every run. A deliberately injected fault produced a failed execution and an email naming the workflow and the node that threw, with the error message. A silent partial batch is the failure mode that matters here, so the run fails loudly rather than delivering less than it should.
Where it fits
- Research and list-building batches where a wrong row costs a campaign.
- Ongoing sourcing with a fixed quality bar and a reviewer who should only see exceptions.
- Any hand-off where "looks complete" has been mistaken for "is complete".