Kestra

Kestra is an open-source workflow orchestration platform for running scheduled and event-driven automations and data pipelines. Workflows are defined as code (YAML) and executed by workers, with a web UI to manage executions, logs, and operational controls.

Key Features

• Workflow-as-code in YAML with reusable tasks, inputs/outputs, variables, retries, and error handling
• Event-driven triggers (e.g., schedules and external events) and dependency-based orchestration
• Web UI for authoring, running, and monitoring executions, including logs and execution history
• Built-in versioning, namespaces, and environment separation to manage many workflows safely
• Rich integrations via plugins (e.g., databases, cloud services, messaging, and scripts) to run end-to-end pipelines
• Runtime controls such as concurrency limits, timeouts, backfills, and manual re-runs
• Operational features including RBAC/tenancy options (edition-dependent), auditability, and API-based automation

Use Cases

• Orchestrate ELT/ETL pipelines across SQL engines, object storage, and transformation tools
• Automate ops tasks (batch jobs, file processing, notifications) on schedules or events
• Coordinate multi-step application workflows (webhooks, queues, microservices) with retries and observability

Limitations and Considerations

• Some enterprise features (e.g., advanced governance/security/SSO) may require paid editions depending on deployment needs.

Kestra is well-suited for teams that want an orchestrator that is both developer-friendly (workflow-as-code) and operations-friendly (strong UI and execution controls). It fits data engineering and general automation scenarios where reliability, visibility, and integrations matter.