From a page to a structured field: an LLM data pipeline

What must be decided before a parser touches the page?

Decide the entity and the fields before you look at a template. A product might need an identifier, a name, a price, a currency, an availability flag, a seller, a region, a source URL, and an observation time. A notice might need an authority, an effective date, and a status. If a field is not on that list, the parser does not invent it. This list is also the minimization rule. Personal names, account ids, and free-text stories stay off unless a reviewed purpose put them there.

Decide the channel in the same pass. If the publisher offers an API or a licensed file with those fields, use it. Page parsing is the residual path for material that is lawfully available only as a rendered page. Adding a domain is a governance event, with an owner and a review date. MetaEdge can route the request and return the response the template allows. The template does not become more permissive because a demo needs one more column. Scope stays fixed.

Decide failure before success. A missing price is not zero. An unparsed date is not today. A page that returns a challenge is not an empty catalog. Each of those outcomes needs a status the downstream system can branch on. Teams that only specify the happy JSON will watch a model fill the gaps with fluent defaults. Write the nulls, the enums, and the stop conditions into the schema first. A happy path alone is not a complete field specification.

How does extraction differ from summarization?

Extraction copies a value that is on the page into a typed slot. Summarization writes a new sentence about the page. The first can be checked against the source string. The second cannot, except by another reading. For prices, dates, identifiers, and statuses, extraction is the right job. A summary may explain a notice after the fields exist. It is a poor place to hide the price. If the only artifact is a paragraph, the next system will parse the paragraph, which is a worse parser than the one you skipped.

Keep the evidence span. For each extracted field, store the raw substring and, when you have it, a stable pointer into the page. A reviewer can then see that 19,90 became 19.90 EUR and not 1990. Without the span, a dispute becomes a matter of taste. With the span, it becomes a diff. This is ordinary data engineering, and it is also how an agent citation stays honest. The model quotes the field. The field points at the span. The span points at the URL and the time.

Do not ask a model to extract and summarize in one call if you need a number you will compute on. The attractive path is one prompt that returns prose plus a JSON blob. The blob then drifts whenever the prompt drifts. Put deterministic parsers on the fields that have a grammar, and use a model only where the page is language rather than a value. When you do use a model, validate its output against the schema and reject the record on failure. A pretty object that fails validation is not a record.

Which normalizations stop silent errors across markets?

Numbers, dates, and money are where silent errors live. A comma may be a decimal mark or a thousands separator. A date written as 10/06/2026 is not the same day in every locale. A price may include tax in one country and exclude it in another. Normalize only after you know the locale of the page, and store the original string beside the canonical value. If you cannot tell, leave the field unknown. Guessing a decimal place at volume will train every downstream chart to trust a fiction.

Units and identifiers need the same discipline. A pack of six is not the same offer as a single unit at a similar price. A seller sku is not a global product id. Colors and sizes that look alike across languages may be different variants. Entity resolution should refuse a merge when the key is weak. The cost of a missed merge is a duplicate row. The cost of a bad merge is a price series that compares two products and calls the difference a market move.

Region stays on the record through normalization. A residential or datacenter route may have been required to see the local page, but the route is not the currency. Write both. MetaEdge can place the request in the geography the template names and return the response. The parser still has to record what the page showed. A normalized euro value without a region and a tax flag is not ready for an assistant that will compare offers across borders. Write the tax flag too.

Where should a language model sit, and where should it stay out?

A model is useful when the page is prose: a disruption notice, a policy paragraph, a product description that must be classified into a small set of labels. Give it the relevant text, a closed label set, and a schema. Ask it to abstain when the text does not support a label. A model is a poor calculator and a poor clock. It should not add tax, convert currencies, or decide that an undated page is current. Those are functions with tests. Put them outside the prompt.

Keep the model off the fetch. It does not choose domains, routes, or rates. A planning loop that can edit the crawler will expand scope the moment a field is empty. The fetch tool takes a source key from the allowlist and returns bytes plus a status. The parse stage is separate. MetaEdge routes and collects within the scope it is given. The language model sees fields, not the right to widen the crawl. That separation is the difference between a pipeline and an unreviewed browser.

Evaluate the model on the field, not on fluency. Sample pages where the template changed, where the value is missing, and where two locales format the same number differently. Score exact match on identifiers and a tolerance you have written down on prices. A high score on a summary rubric can hide a systematic decimal error. Publish the failure cases beside the success cases. The NEST plan and any later GPU plan do not grade this test. The fixtures do.

How do you keep the raw observation next to the derived field?

Store two layers. The raw layer holds the URL, the fetch time, the status, a hash of the body, and a retained sample or pointer under a short retention rule. The derived layer holds the typed fields, the parser version, and the raw substring for each field. They share a document id. When a parser changes, you can reprocess from the raw sample without pretending the new parse was what you observed last month. When a lawyer asks what you kept, you can point at the retention rule instead of a bucket that grew without one.

Hashes make drift visible. If the body hash changes and the fields do not, the parser may be blind to a template move. If the fields change and the hash does not, someone edited downstream. Alert on both. A pipeline that only watches HTTP 200 will stay green while prices come back empty. Field completeness, schema violations, and sudden shifts in value belong on the same dashboard as latency. MetaEdge can return the response and the route. Your validator decides whether that response became a record.

Deletion has to cross both layers. If a source leaves the allowlist, the raw sample and the derived fields go with it, and so do index entries an agent might still cite. A document id makes that search possible. Training manifests that pointed at the id can be rebuilt without those examples. This is not a claim that a previously trained model forgets. It is the control that stops the next run from repeating the inclusion. Provenance that cannot be queried is a paragraph in a policy, not a pipeline.

What does a production record look like when it reaches retrieval?

A production record is small and explicit. It carries the entity key, the typed fields, the locale, the region of the request, the route type, the source URL, the observation time, the parser version, and a status. Optional text for explanation sits in a separate field, marked as text. Retrieval can filter on age and on status before the model sees anything. An agent tool returns this object, not a page. If a required field is null, the status says why, and the agent is not free to fill it.

The same record can feed more than one store if the purpose tag travels with it. A live index for RAG keeps current rows. A static snapshot for evaluation freezes a version. A training mix takes only rows whose purpose and rights review allow it. Copying the table because it is clean is how a price series becomes training data nobody approved. MetaEdge can deliver the structured payload. The customer's registry decides which door it walks through. Purpose stays attached.

The test of the pipeline is a question a person can audit. Which page, observed when, through which route, produced this price, and which parser turned the string into a number? If the system can answer that, an assistant can cite it and a training run can include or exclude it on purpose. If it cannot, the model is summarizing a page nobody can reconstruct. Structured fields are how web data stays inspectable. They are not a claim that every page on the web is yours to take.

FAQ