Data pipeline definition

A data pipeline is a system that carries data from a source to a destination, applying a fixed sequence of steps such as validation, deduplication, and reformatting on the way. Most pipelines discover that a source changed by polling it on a schedule. In Sanity, a Function or a document webhook fires on the create, update, or delete event itself, and a GROQ filter decides which changes are worth waking the pipeline for, so the downstream system receives a filtered change event instead of a full re-scan.

What are the parts of a data pipeline?

A data pipeline has three structural parts: one or more sources, a set of processing steps, and one or more destinations. A source can be an application, a device, a database, or a continuous stream, and data either gets pushed into the pipeline or pulled out of the source by an API call, a webhook, or a duplication process. The processing steps do the work in between. AWS lists the common ones as sorting, reformatting, deduplication, verification, and validation.

The destination is where the data lands, usually a data warehouse, a data lake, a search index, or another application. It is sometimes called a data sink. Steps in a data pipeline also carry dependencies, which are the conditions that have to be satisfied before a step can run. A technical dependency might be waiting for a queue to drain. A business dependency might be waiting for another team to verify a batch of figures before it is allowed downstream.

Broadly, data pipelines fall into four commonly named kinds: batch processing pipelines, streaming pipelines (which IBM also calls event-driven architectures), data integration pipelines, and cloud-native pipelines.

What is the difference between a data pipeline and an ETL pipeline?

An ETL pipeline is a specific kind of data pipeline, so the relationship only runs one way: every ETL pipeline is a data pipeline, but not every data pipeline is ETL. ETL stands for extract, transform, and load, and as AWS puts it, an ETL pipeline is "a special type of data pipeline."

The dividing line is transformation. Changing the data is definitional to ETL and optional to a data pipeline in general. IBM states it under its own heading: data pipelines do not always require data transformation. Some pipelines simply extract records from a source and load them somewhere else untouched, which is still a data pipeline.

ELT (extract, load, transform) is a reordering of the same three steps rather than a different category. An ELT pipeline loads raw, often unstructured data straight into a data lake and applies transformations later, just in time for analysis. That order suits destinations with enough compute to transform on read, and it keeps the original records available in case the transformation logic needs to change.

Is a content pipeline the same thing as a data pipeline?

No. A content pipeline and a data pipeline share the word "pipeline" and very little else. A content pipeline is an editorial term: it describes the stages a piece of work passes through on its way to being published, typically idea, brief, draft, review, approval, and publish. It is a process model for human and agent work, and the thing moving through it is a piece of content in progress.

A data pipeline is a transport and transformation system. The thing moving through it is records, and the steps are automated operations on those records rather than decisions made by people.

The two do meet at the edges. A content pipeline that ends in a publish event can be the trigger for a data pipeline that pushes the published record into a search index, a translation service, or a product feed. But confusing them leads to real design mistakes, such as modeling editorial review states as pipeline stages in an orchestration tool, which makes the review process brittle and hard for editors to see.

Batch or streaming: which kind of data pipeline should you use?

Batch and streaming are properties of a data pipeline rather than two separate species of it, and IBM notes that data pipelines support both. Choose based on how fresh the destination needs to be and what shape of compute you can afford.

A batch data pipeline runs infrequently, often during off-peak hours, and needs a lot of computing power for a short window. Batch processing has deep roots here: IBM points to MapReduce, patented in 2004 and later built into open-source systems including Hadoop, CouchDB, and MongoDB. A streaming data pipeline runs continuously and needs less computing power at any one moment, but it depends on reliable, low-latency network connections. Streaming pipelines usually involve a message broker such as Apache Kafka, and often a change data capture (CDC) tool to detect source changes as they happen.

It is also worth separating the pipeline from the orchestrator. The orchestrator schedules, sequences, and retries the steps; the pipeline is the thing being orchestrated. An academic survey of data pipeline tooling treats ETL/ELT tools, ingestion tools, orchestration, and workflow management as distinct tool categories.

Why do data pipelines cost so much to maintain?

Data pipelines cost so much to maintain because they break when something upstream changes, and someone has to notice and fix each break. A schema changes, an API deprecates a field, a credential rotates, or a source starts sending a slightly different date format, and the pipeline either fails loudly or, worse, keeps running and delivers wrong data.

The scale of that upkeep shows up in vendor research. Fivetran's 2026 enterprise data infrastructure benchmark report, published on March 26, 2026 and based on a survey of more than 500 senior data and technology leaders at organizations with over 5,000 employees, reports that data teams spend 53% of engineering time on maintenance. Fivetran sells data movement tooling, so read the number as a vendor-published figure rather than a neutral industry constant, but the direction it points in matches what pipeline owners tend to describe.

One structural way to reduce that surface is to stop polling. A pipeline that asks "has anything changed?" on a timer has to re-scan, diff, and guess. A pipeline fed by change events only does work when work exists. Sanity is a Content Operating System for the AI era, and its automation layer is built for exactly that handoff: Functions run on Sanity's infrastructure in response to content changes and can call the full Sanity APIs, not just the document that triggered them, while GROQ projections control which sections of content get passed downstream. Document webhooks fire on every create, update, or delete with GROQ filtering and custom payloads, and the Webhooks API exposes delivery attempts and messages so you can inspect what was actually sent.

Last updated: September 22, 2026