Prefect — Workflow Orchestration

Prefect 3.3.2 is the orchestration layer for all data ingestion and transformation workflows. It schedules, monitors, and re-runs pipelines that pull data from Rwanda’s health source systems into the data warehouse.

Health Source IngestionIngests data from HMIS, IDSR, eFICHE, HIV Tracker, Immunization, and Cancer Registry.

dbt TransformationsTriggers dbt transformation runs after each ingestion to keep the warehouse current.

Automated ReportingGenerates the IDSR Bulletin and Rwanda Health Intelligence Newsletter on schedule.

Spark Job SubmissionSubmits distributed Spark jobs via the Livy REST API as part of larger pipelines.

Quick Reference

Attribute	Value
Version	Prefect 3.3.2
Python	3.11
Work pool	default
Config file	`apps/analytics/prefect/.prefect/prefect.yaml`
Workflows root	`apps/analytics/prefect/.prefect/workflows/`
Timezone	Africa/Kigali