data-pipelines

Apply Data Pipelines Pocket Reference practices (James Densmore). Covers Infrastructure (Ch 1-2: warehouses, lakes, cloud), Patterns (Ch 3: ETL, ELT, CDC), DB Ingestion (Ch 4: MySQL, PostgreSQL, MongoDB, full/incremental), File Ingestion (Ch 5: CSV, JSON, cloud storage), API Ingestion (Ch 6: REST, pagination, rate limiting), Streaming (Ch 7: Kafka, Kinesis, event-driven), Storage (Ch 8: Redshift, BigQuery, Snowflake), Transforms (Ch 9: SQL, Python, dbt), Validation (Ch 10: Great Expectations, schema checks), Orchestration (Ch 11: Airflow, DAGs, scheduling), Monitoring (Ch 12: SLAs, alerting), Best Practices (Ch 13: idempotency, backfilling, error handling). Trigger on "data pipeline", "ETL", "ELT", "data ingestion", "Airflow", "dbt", "data warehouse", "Kafka streaming", "CDC", "data orchestration".