A self-paced, practical course for data engineers, analytics engineers and developers responsible for moving and transforming data between systems. Seven lessons cover ETL versus ELT and pipeline architecture, batch and streaming ingestion, change data capture and incremental processing, transformation design, orchestration and scheduling, data quality and observability, and security, cost and operations. Each lesson ends with an exercise that builds a working pipeline step by step.
What you will learn
- Choose between ETL and ELT and design pipeline architecture
- Implement batch and streaming ingestion
- Use change data capture and incremental processing
- Design maintainable, idempotent transformations
- Orchestrate pipelines with dependencies, retries and backfills
- Test data quality and monitor pipelines
- Secure pipelines and control cost
Lessons
- ETL, ELT and Pipeline Architecture
- Batch and Streaming Ingestion
- Change Data Capture and Incremental Processing
- Transformation Design
- Orchestration and Scheduling
- Data Quality and Observability
- Security, Cost and Operations
Self-paced course from Tech Skills Library. Educational content only; it is not an accredited qualification, licence or credential.
Lessons
ETL, ELT and Pipeline Architecture
Batch and Streaming Ingestion
Batch ingestion Data is extracted at intervals, such as hourly or daily. Simpler to build and operate; suitable when slightly stale data is acceptable. Streaming ingestion Events are captured continuously through message brokers or streaming services and processed within seconds. Needed for real-time dashboards, alerting and operational use cases. API ingestion Handle pagination, rate limits, […]
Change Data Capture and Incremental Processing
Why incremental Reloading entire tables wastes time and money as data grows. Approaches Timestamp-based: select rows where updated-at is greater than the last watermark. Simple but misses deletes and depends on reliable timestamps. Log-based change data capture: read the database transaction log to capture inserts, updates and deletes with low impact on the source. Snapshot […]
Transformation Design
Layers Staging models clean and rename; intermediate models join and apply business logic; mart models present data for consumption. Idempotent transformations Rebuilding a model should give the same result. Avoid logic that depends on run time unless explicitly parameterised. Incremental models For large tables, process only new partitions and merge results. Modularity Small, well-named models […]
Orchestration and Scheduling
Orchestrators Orchestration tools define pipelines as directed acyclic graphs of tasks with dependencies, schedules, retries and alerting. Good practices Make tasks small and idempotent. Use retries with backoff for transient failures. Use sensors or event triggers rather than fixed waits. Parameterise tasks by date to support backfills. Keep secrets out of pipeline code. Backfills Rerun […]
Data Quality and Observability
Quality tests Schema checks, not-null and uniqueness on keys, referential integrity, accepted values, volume checks against expected ranges and reconciliation with source totals. Where to test Test at ingestion to catch source problems and after transformation to catch logic errors. Observability Track run durations, row counts, freshness, failures and data quality results over time. Use […]
Security, Cost and Operations
Security Use least-privilege service accounts, encrypt data in transit and at rest, store credentials in a secrets manager and mask or exclude personal data that is not needed. Cost Monitor compute and storage costs per pipeline, use incremental processing, schedule heavy jobs efficiently and clean up temporary data. Operations Use version control, code review and […]