Architecture Pattern
Data Pipeline Architecture (ETL/ELT)
The structured flow that moves data from source systems into a place it can be reliably reported on.
What it is
Data is Extracted from source systems (CRM, ERP, product database), optionally Transformed into a clean, consistent shape, and Loaded into a destination (a data warehouse or reporting database) — ETL transforms before loading; ELT loads raw data first and transforms afterward, which has become more common as warehouses got cheaper to query directly.
When to use it
Whenever a dashboard or report needs to combine data from more than one source system reliably, rather than someone manually exporting and merging spreadsheets each cycle.
Real tradeoffs
- —The pipeline itself needs monitoring — a silently broken pipeline produces a dashboard that looks fine but is quietly wrong
- —ELT is more flexible (raw data is preserved, transformation logic can change later); ETL can be simpler when the destination system can't handle heavy transformation itself
- —Garbage in, garbage out still applies — a pipeline can't fix inconsistent source data, only move it faster
Related
FAQs
Do we need a data pipeline if we only have one data source?
Usually not — a pipeline earns its complexity when multiple source systems need combining. For one source, a direct connection or simple scheduled export is often enough.
Have a project in mind?
Tell us what you're trying to automate or build — we'll reply with next steps, not a sales pitch.