Proposal: deterministic schedule jitter so daily DAGs don't all fire at the same time

[Pebble32]

Hi all,

Not sure what the formatting here is and if there is a template for me to use to apologies if the setup is unclear. I tried to include all relevant information needed.

The problem

In bigger deployments a lot of DAGs end up on the same cron time. @daily always resolves to 0 0 * * *, so every daily DAG fires at midnight at the exact same moment. On a production MWAA environment I work on we had around 26 daily DAGs all firing at 00:00 and it caused task failures from the contention at that boundary.

Right now there is no native way to spread them out. You either:

• hand pick a unique minute for every DAG, which drifts and collides as the number of DAGs grows, or
• hash the dag id into a literal cron string yourself, which works but throws away the @daily intent and is not reusable.

Neither feels like something every team should be reinventing on their own.

What I would like to propose

A way to add deterministic jitter to a schedule. Something like a JitteredCronTimetable, or a jitter option on the existing cron timetables, that offsets each DAG by a stable function of its dag id inside a configurable window (say up to 60 minutes), while keeping the data_interval and logical_date semantics exactly the same.

The important part is that it stays deterministic rather than random, so a given DAG always lands in the same slot and runs stay stable and predictable across scheduler restarts and timetable serialization.

Prior art

I looked and could not find anything that already does this. No native option, and no community plugin that I could find. Other schedulers tend to offer something in this space (Kubernetes CronJobs for example).

Questions before I build anything

1. Is this something you would want in core, or is it better as a community plugin?
2. If core, what shape do you prefer, a new Timetable or a jitter option on the existing cron timetables?
3. Does this need an AIP, or is it small enough to go straight to a PR?

I already have a working product at my current workplace in production and I am happy to bring tests and docs.

[raphaelauv]

from the contention

set airflow pools on your tasks to restrict the contention and if airflow pools are not enough for your use case then you must put in place something for your specific case

[Pebble32]

Thanks for the reply, that is fair and pools are a good call for capping how many tasks run at once.

The gap I am trying to close is a little different though. Pools throttle execution, but every DAG is still scheduled at the exact same instant, so the scheduler still gets the midnight spike and the work just queues instead of spreading out in time. For anything tied to a specific moment (an external API, or a database that is busy right at midnight) spreading the start times helps in a way a concurrency cap does not.

On the "put something in place for your specific case" part: I have actually already built a working version, and I am happy to write it up properly with tests and docs and maintain it going forward. Maintaining it is genuinely not a problem for me, I'm big fan of the project and would love to help. I just think it would help plenty of other deployments that hit the same midnight pile up, which is why I would rather contribute it than keep it local.

Would a small deterministic jitter timetable be welcome, either in core or as a community plugin? Happy to go whichever route you prefer.

[raphaelauv]

there are already many concurrent setting to limit spike

AIRFLOW__CORE__DEFAULT_POOL_TASK_SLOT_COUNT=128
AIRFLOW__CORE__PARALLELISM=32
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG=16
AIRFLOW__CORE__MAX_MAX_ACTIVE_RUNS_PER_DAG=2

if things must start at midnight , then you want your airflow to start the maximum number of things at midnight, if it put pressure on your hardware or external resource then update your scaling or max PARALLELISM

[Pebble32]

Totally fair, and those knobs are the right tools when the work genuinely has to run at midnight.

The one thing I would add: all of those limit or queue execution, but the DAGs are still scheduled at the same instant. And in the case I'm thinking about here is that they are at midnight by accident, not by need. @daily just resolves to 0 0 * * *, so a bunch of jobs that only need to run once a day all land on the same second. Jitter is really about removing that accidental pile up so you do not have to over provision or hand tune pools for a spike that did not need to exist. I thought that a deterministic way of shuttering DAGs around would be a nice quality of life feature.

That said, I get that this may not be something core wants to take on, and that is completely reasonable. I will build it as a small opt in plugin so anyone who hits the same thing can use it. If a maintainer ever feels it would be a welcome addition to core, I am happy to bring a PR with tests and docs. Thanks for talking it through. <3

[raphaelauv]

That was just my opinion, feel free to write a msg on the dev list or open a PR 🙂