Data Pipeline
Move data. Unlock value.
Data Pipeline moves information from your sources through processing stages to the destinations where applications, analytics systems and agents use it. It is the data foundation for Agent Builder, Agent Run and Generative UI.
From source to usable data
A pipeline can include ingestion, validation, transformation, enrichment and delivery. ETL is one type of pipeline; others follow ELT or move data with little transformation. Batch and streaming are two processing approaches within this broader category.
Pipeline stages
| Stage | Responsibility | Example |
|---|---|---|
| Connect | Authenticate and communicate with a source | API, database, file store or stream |
| Ingest | Retrieve records, documents or events | Fetch newly available records |
| Validate | Check schemas and required values | Reject or quarantine malformed records |
| Transform | Standardize and map information | Normalize units and timestamps |
| Enrich | Add useful context | Associate a point with equipment and facility |
| Deliver | Write to a supported destination | Database, object storage, index or event stream |
| Observe | Track freshness, throughput and failures | Detect a stalled ingestion job |
Processing modes
| Mode | Typical use |
|---|---|
| Scheduled batch | Nightly reports or periodic synchronization |
| Incremental polling | Fetching records changed since a saved cursor |
| Event-triggered | Processing a newly uploaded document |
| Streaming | Continuously processing incoming measurements or events |
| Backfill | Loading or repairing historical periods |
Documents: extraction and validation
PDF, Word and spreadsheet content is extracted into structured records and then validated as a separate step. A table can be extracted successfully and still need its dates, amounts, identifiers and confidence checked before its rows are accepted.
Built for reliable delivery
Apache Kafka can provide the event streaming layer, with Kafka Connect supplying reusable import and export connectors. Around it, Data Pipeline provides the parsing, validation, transformation, orchestration and storage writing that each source needs, including:
- Configurable connectors.
- Bounded concurrency and rate-limit handling.
- Saved cursors and retries.
- Deduplication.
- Schema versioning.
- Historical replay.
- Source lineage.
Typical data
- Operational measurements, metadata and rule detection results.
- PDF, Word and spreadsheet extraction into structured records.
- Market prices and company financial data.
- Business-system synchronization.
Make distributed information available in a consistent, timely and traceable form.