Provenance and quality: freshness, coverage, bias

What does provenance need to contain to be useful later?

Provenance is the smallest set of facts that lets a later reader reconstruct an observation. At minimum, store the canonical URL, the fetch time, the HTTP or render status, the parser version, the language detected on the page, the region of the request, and the route type. Store a hash of the body you kept and the raw substring behind each derived field. If a person approved the domain, store the identifier of that decision. A folder name with a date is not provenance.

Separate clocks on purpose. observed_at is when your collector saw the value. published_at is the date the source printed, when it printed one. effective_at is when a rule takes effect, when the page says so. Mixing them makes an assistant call an old notice new because it was fetched this morning. The record should make the confusion expensive. Retrieval and agents should be able to filter on the clock that matches the question, not on whichever timestamp a loader wrote first.

Provenance has to be queryable. If a source is later judged out of scope, you should find every derived row and every index entry that used it. Training manifests should list example ids that point back to these rows. MetaEdge-produced records can carry the collection facts. The customer's registry still decides retention and who may read the trace. A provenance paragraph that lives only in a policy file will not answer an incident. The stored trace has to answer the incident.

How should freshness be defined at the field, not the file?

A file can be new while a fact inside it is old. Define a bound per field. A price may need to be younger than an hour. A product description may tolerate a week. A statute may tolerate longer, provided the record shows the observation time and the source's own date. The application then knows which claim it may state and which claim it must qualify. One latency number for the platform describes the network. It does not describe the data.

Freshness includes the failure. If a refresh misses the bound, the last value stays available only with its age attached, or the claim is dropped. It is not promoted to current because the job ran. Conditional requests, hashes, and priority queues can keep cadence proportionate to volatility without hitting every URL every minute. Backoff remains part of the design. MetaEdge can route the attempt the template allows and return the status it got. The template owns the bound. Record the miss.

Test freshness with a clock you control. Freeze a fixture, move the page, and fail the fetch. The assistant should cite the old observation as old, or abstain. Teams that only demo a green collector will not see this. Agents are especially exposed, because they speak in the present tense by default. A tool contract that refuses out-of-bound fields is the practical definition of real time. It is stricter than a marketing line, and it is the one users can check.

What does coverage mean when the web is not a population?

The public web is not a census. Coverage means the share of the sources you named, in the regions you named, that returned a valid field inside the bound. It does not mean a percentage of the internet, and it does not mean a share of a market. Report domains in scope, domains reached, fields complete, and fields missing. If a retailer has no public price in a country, that absence is a finding. Filling it from another country is a coverage error, not a completion.

Language and geography are part of coverage. A corpus that is almost entirely English does not cover Italian, Spanish, German, French, or Portuguese pages by adjacency. A datacenter route in one region does not cover the household view in another. State these limits in the quality report next to the counts. Buyers should treat a vendor map that shades every country the same as a drawing, not as a measurement. Ask which of your URLs were actually fetched, and from where.

Coverage changes when the source changes. A template move can zero a field while HTTP stays 200. A site can split one URL into many variants. Track completeness over time, not only on the day of the proof of concept. MetaEdge can keep collecting the scoped list. The customer decides whether the remaining coverage still supports the decision. If it does not, the honest output is a smaller claim, not a smoother chart. Shrink the claim before you smooth the chart.

How does collection bias enter before the model is trained?

Bias enters when easy pages crowd out the pages you meant to learn from. Large sites, syndicated copies, and printer-friendly URLs repeat the same text. Without deduplication, frequency becomes importance. Boilerplate, if left in, teaches navigation instead of subject matter. A single region teaches that region's prices and spellings as if they were universal. None of this requires a malicious crawler. It requires an unscoped one. The sampling plan is the countermeasure, written before the crawl and checked after it.

Selection bias also sits in what you refuse. Dropping small sites because they are harder to parse narrows the voice of the corpus toward whoever publishes clean templates. Keeping only pages that match a keyword list narrows it toward that list. Write the inclusion rule and the exclusion rule, and sample the tails. Domain reputation alone is a weak proxy: a known domain can host a thin tag page, and a small domain can host a careful specification. Report what you excluded and what you still do not know.

Do not invent a market-wide quality score. A pipeline can measure its own emptiness, duplication, and concentration. It cannot, from those meters, announce how biased the web is or how large a category has become. Hand the measured mix to the people who know the users of the product. If the users are in markets the sample barely touches, the model will not acquire those conventions by accident. Training on the sample you have is a choice. Provenance makes the choice visible. It does not launder it.

Which checks catch a template change that still returns HTTP 200?

Watch the fields, not only the status line. Alert when completeness drops, when a numeric field becomes text, when values jump beyond a threshold you wrote down, or when the body hash changes while the extracted price stays null. Keep a canary set of URLs with known shapes and run it on the same cadence as production. A green collector and an empty price column is a data incident. Treat it as one. The page did not fail. The contract between parser and page did.

Pin the parser version on every row. When you ship a fix, new rows show the new version, and you can compare distributions before and after on the same URLs. Reprocessing a raw sample is legitimate. Silently rewriting history is not. If the raw sample has expired under its retention rule, say so and collect forward. MetaEdge can return the response your template requested. Your regression set decides whether the parser still deserves to write a record. Keep the version visible.

Sample with a person on a schedule, not only when an alarm fires. A hundred rows across locales will show a decimal error, a merged variant, or a banner that the model would have trusted. Record the disagreements. Automated checks catch the breaks you predicted. Human review catches the breaks you did not name. Neither replaces the other's job. A quality program that is only a dashboard will miss the row that looks valid and means the wrong thing. Review it.

How should an application behave when quality falls below the bar?

Below the bar, the application narrows the claim. It can show the last observation and its age, omit the field, or send the case to a person. It should not let the model invent a current value to keep the sentence smooth. This behavior is a product requirement, and it should be tested. RAG filters should drop or demote passages that miss the freshness or completeness rule. Agents should receive a status, not a hint to try a cleverer fetch.

Tell the user what you know. A sourced price with a timestamp is more useful than a fluent range with no source. If coverage is thin in a region, say that the view is limited to the sources you reached. Do not translate a partial crawl into a market conclusion. MetaEdge's role ends at collection and routing inside the scope you set. The sentence that reaches a customer, a regulator, or an internal dashboard is still yours, and it should be no stronger than the record.

Quality work is ongoing because the web moves. Revisit bounds, allowlists, and parsers when sources change, and keep the provenance that makes those revisits possible. The NEST plan for 2026 may widen household supply. It will not decide representativeness. Storage and GPU plans will not decide it either. A team that can answer where a number came from, how old it is, and what it fails to cover is in a position to use web data. A team that cannot is in a position to pause.

FAQ