Keeping data lineage when intelligence products combine several sources

An intelligence data product can cite every input and still leave its customer unable to explain how the final conclusion was reached. The missing information is often the relationship between the sources: which version was used, what transformation was applied and who took responsibility for the resulting assessment. That relationship is the practical purpose of data lineage.

For a commercial supplier, lineage is part of product quality. It helps a customer investigate a disputed result, understand a revision and decide whether earlier outputs need to be revisited. It also makes the supplier's own contribution visible. A company selling analysis should be able to distinguish that work from the underlying data it acquired.

A source list answers only the first question

A bibliography or source panel identifies material associated with a report. It may not establish which sources support a particular statement, whether they were independently obtained or whether one merely repeats another. It may also point to a live page that has changed since the original assessment.

A useful lineage record goes further. It connects a delivered result with identifiable inputs and the activities that produced it. The depth can be proportionate to the product. A short market summary may need a clear source-to-claim account; a continuously updated analytical dataset may need machine-readable relationships and version identifiers.

W3C's PROV overview describes a family of models and representations intended to exchange provenance information across systems. Its core concern is information about the entities, activities and agents involved in producing something. That provides a vocabulary for the problem without prescribing a single commercial implementation.

The important purchasing question is whether the customer can follow the parts of that history that matter to its decision. A platform can display an elaborate source graph while omitting the exact input version or analytical change that explains the result. Visual complexity is not a substitute for the relevant evidence.

Keep observations, transformations and judgments distinct

A derived product often contains several kinds of contribution. One source supplies an observation or published statement. A processing step converts or combines information. An analyst then interprets the result. If those layers are compressed into one unqualified field, a customer may mistake an inference for a directly observed fact.

The PROV primer distinguishes entities, activities and agents with responsibility for those activities. It also treats versions as identifiable things. Applied to a commercial product, this encourages a clear account of the input, the transformation and the person or organisation responsible for the output.

Consider a hypothetical industry dataset that combines company announcements with procurement notices to summarise industrial expansion. An announcement describes intended capacity; a notice records a purchasing event; an analyst infers a possible relationship. The final product should preserve those different roles. The existence of the two source documents does not make the inferred relationship a reported fact.

A usable record can identify the relevant source versions and label the analytical step without exposing proprietary code. The buyer needs enough information to assess the meaning and limitation of the conclusion, rather than every detail of the supplier's internal implementation.

Trace the part of the result that can change

Lineage becomes especially valuable when a source is corrected. The customer needs to know which delivered outputs relied on the earlier version and whether the correction affects their conclusions. A generic list of sources for an entire database makes that task unnecessarily expensive.

The supplier can instead preserve relationships at the level appropriate to the product: a report section, a derived indicator or an individual record. The choice should reflect the likely review question. Excessively fine detail creates maintenance work; excessively broad links make it difficult to identify affected outputs.

A revision can also originate inside the supplier. A new calculation method, category definition or analyst amendment may change the output even when the sources remain the same. The product history should distinguish that change from a newly acquired fact. Otherwise customers may interpret a processing revision as a development in the underlying industry.

Our guide to imagery revisit and delivery latency explains the related timing problem. A newly generated result can concern an earlier observation. Lineage helps preserve that relationship instead of allowing the latest processing date to stand in for the age of the evidence.

A shared format helps only if meaning survives

The OGC Observations, Measurements, and Samples standard provides a conceptual schema for observations and the features and samples involved in them. Its stated purpose includes exchanging descriptions of observation acts and their results across communities. It addresses a narrower observational model than a complete commercial intelligence dossier.

A supplier integrating such data still needs to explain what the imported values mean in its own product. A field can be successfully transferred while losing its qualification, unit or relationship to a sample. Technical interoperability should preserve the interpretation required by the customer, not merely the ability to open the file.

The acceptance discussion can use a real exported example. Can another authorised reader identify the source version, distinguish an observed value from a derived one and understand the relevant qualification? If the explanation exists only inside the vendor's interface, the customer should know that the exported product has a narrower capability.

This is also a useful boundary for systems integrators. The parties can agree which provenance information each handover must retain and who is responsible when a receiving system cannot represent it. The missing information then becomes an explicit integration issue rather than an unexplained analytical discrepancy.

Provenance does not establish truth or permission

A complete history can show exactly how an unreliable conclusion was produced. It improves inspectability; it does not remove the need to assess source quality, analytical method and the scope of the claim. The supplier should avoid presenting a traceable record as if traceability alone validates the result.

The same applies to rights. Knowing where data came from does not establish permission to redistribute them or every derivative. Provenance can help a commercial team identify which terms need review, but the permission comes from the relevant licence or other applicable basis. Those are separate parts of the delivery record.

Our coverage of GAO's commercial-space data adoption findings explains why technical availability and customer use can be different questions. A product that combines several sources should retain the information needed to identify those boundaries without assuming that one source's permissions apply to the whole output.

Restricted inputs also require a proportionate explanation. A supplier may be unable to provide full source material to every customer. It can still describe the scope of the evidence accessible for review and the responsibility it takes for the derived result. The limitation should be part of the offer.

Price the reviewable product

Maintaining meaningful lineage requires work: stable identifiers, version histories, documented transformations and a process for handling corrections. A supplier can include that work in its product rather than treating every customer query as a bespoke investigation. The resulting service may justify a different price from a basic data feed.

A procurement team should establish which review functions are included. Does the customer receive a source account with each delivery? Can it reconstruct an earlier version after the subscription changes? Who identifies downstream outputs affected by a correction? Answers to those questions reveal the practical value of the lineage claim.

A company profile such as BlackSky supplies market context for a data business. The purchasing evidence is more specific: a sample result whose important sources, transformations and responsibilities can be followed through an ordinary customer review. That is the point at which lineage becomes a usable commercial capability.