Old Data Pipelines, New AI Experiments

Before machine learning became an everyday product term, we built data systems that processed enormous volumes of ordinary events. Purchases, searches, readings and status changes moved through pipelines, accumulated into measures and appeared in dashboards where someone made a decision.

The systems were less conversational than today’s AI, but they taught lessons worth carrying into new experiments.

The first is that data never arrives without a history. A field has a source, a transformation and a reason it exists. When that history is lost, numbers become difficult to challenge. A total may be correct according to the pipeline and wrong for the question a person thinks they are asking.

Generative systems make provenance more difficult and more important. An answer can combine retrieved documents, model behaviour and instructions into a single fluent paragraph. The reader sees the paragraph, not the joins. An experiment should preserve enough context to reconstruct the path: which sources were available, which version of the prompt and model was used, and what transformations occurred before the result appeared.

This does not mean logging every sensitive input forever. Retention creates privacy and security risks. It means designing traceability proportionate to consequence. A creative writing aid may need little history. A tool that summarises clinical guidance needs strong source links, versioning and carefully governed records.

The second lesson is that averages conceal boundaries. A pipeline processing 1.5 billion observations can appear healthy while a small category fails consistently. A model evaluation can show high overall quality while performing poorly for a language, document type or unusual case. Sampling should therefore follow difference, not only volume. Look specifically at the cases the system is most likely to flatten.

Third, silent failure is more dangerous than visible failure. Traditional pipelines may drop messages, reuse stale data or substitute defaults. Generative tools can fail more gracefully in appearance: they produce a coherent answer even when evidence is weak. Observability must include uncertainty and absence. Did retrieval find an authoritative source? Did the model cite something that actually supports the statement? Has the underlying collection changed since evaluation?

These lessons might sound like reasons to avoid experimentation. I see them as ways to experiment more freely. A small test with clear boundaries is easier to stop, compare and learn from. If the source set is versioned, a prompt change can be evaluated honestly. If failure categories are recorded, an imaginative prototype can reveal where the capability breaks without being mistaken for a finished service.

One useful experiment is to build a question-answering tool over a limited archive. Begin with fifty documents whose ownership and quality are understood. Require every answer to point back to passages in those documents. Include an explicit “insufficient evidence” response. Ask subject experts to create difficult questions, including ones the archive cannot answer.

The interesting findings may not concern the model at all. The archive may contain contradictory guidance. Documents may lack dates or owners. People may use the same term for different concepts. The AI interface reveals weaknesses in the information system it sits upon.

That pattern also appeared in earlier data work. A new dashboard did not merely report the organisation; it exposed disagreement about definitions. A pipeline did not simply automate transfer; it made inconsistent processes collide. Emerging tools often create value by revealing the structure around them, even before their intended feature is ready.

The past cannot provide a blueprint. Today’s models introduce distinct questions about generated content, embedded training data and unexpected capabilities. Old controls can create false confidence if copied without understanding. A checksum cannot tell us whether an explanation is fair, and a lineage graph cannot decide whether a use is appropriate.

Teams can make this practical with an experiment log that records more than successful prompts. Include source versions, failed cases, unexpected outputs, reviewer disagreement and the decision each trial informs. Over time, the log becomes evidence of where the capability is stable and where enthusiasm has been doing more work than the model.

Still, discipline and wonder are compatible. We can ask what a new model makes possible while tracking the evidence that shaped its output. We can enjoy surprising results while testing the cases that resist them. We can prototype quickly while defining how the prototype will be contained.

Experience should not close the door on unfamiliar technology. Its best use is to provide handrails: enough structure to explore the edge without pretending the ground beyond it is already mapped.