Real-time operations dashboard for Apache Airflow

Why real-time matters

Critical batch windows — overnight settlements, market open, payroll cutoff — need NOC-style screens that update every few seconds. Airflow's UI auto-refresh queries the metadata database and becomes expensive at scale. A dedicated observability stack (Prometheus + Grafana) handles this with sub-second latency at any workload size.

The reference architecture

Standard panels we ship

• Live job count: queued, running, succeeded, failed in the last 60 minutes
• SLA breach ticker: DAGs missing SLA right now
• Worker pool saturation by queue
• Success-rate heatmap by hour-of-day × day-of-week
• Top-N longest-running tasks with trend vs. 7-day baseline
• Cluster-wide latency from queue → start
• Per-region split for multi-region Airflow deployments

What BatchFoundry delivers

• statsd_exporter + Prometheus + Grafana stack as code (Helm charts or Terraform)
• Pre-built dashboard panels — importable JSON, version-controlled
• Custom DAG callback library emitting business-metric events alongside infrastructure metrics
• Optional NOC kiosk view for operations floor display

Pitfalls we plan around

• Prometheus retention vs. cardinality — don't tag by dag_id if you have 10k+ DAGs
• XCom-based metric exfiltration — use callbacks, not XCom reads
• Authn / authz on the Grafana dashboard — lock to corporate SSO, not open by default