Most model training stories still begin with a dataset: a folder, a dump, a date. That picture is useful, and it is incomplete. A large share of the questions companies now ask are about the public web as it stands this morning — a price, a policy, a product page, a filing — not about a snapshot frozen for a benchmark.
Live web data does not make a model wiser by itself. It changes the job of the system around the model. Someone has to decide which URLs are in scope, how fresh a copy must be, what to keep when a page disappears, and which sentences are allowed to be repeated. Those decisions are operational. They are also the part a buyer can actually inspect.
Time belongs in the record
A sentence without a time is easy to misuse. If a page said one thing in March and another thing today, both statements can be true of the page and false as a timeless fact. A usable record carries the URL, the time it was fetched, and enough of the surrounding text to see what changed. MetaEdge describes its current work, since 2023, as infrastructure for that kind of public-web collection, not as a model and not as a guarantee that every page on the internet is available.
Provenance is a field, not a slogan
Provenance means you can point to where a value came from. For a public page, that is usually the URL, the fetch time, and the extractor that turned a paragraph into a field. It is not a claim that the page was correct. Pages are written by other people, and they can be incomplete or out of date. The honest product is a traceable copy, plus a clear statement of what was not collected.
What a contract should refuse
A serious supply agreement draws a line around personal accounts, private dashboards, and anything the customer's own instructions do not allow. It also refuses invented coverage numbers. 'The whole web' is not an operational description. A market, a set of page types, a freshness window and a failure log are. If those four are missing, the rest of the brochure is not yet a service.
Sources
METAEDGE
What does your system need to see?
Tell us the workload, target markets and collection cadence. We’ll discuss the right data and access layer.
Talk to us→