Run Queue

Terragrunt’s “Run Queue” is the mechanism it uses to manage the run order and concurrency when running OpenTofu/Terraform commands across multiple Terragrunt units. This is particularly relevant when using the run --all or run --graph commands.

At its core, the Run Queue relies on a Directed Acyclic Graph (DAG) built from the dependencies defined between your Terragrunt units. These dependencies are typically established using dependency or dependencies blocks in your terragrunt.hcl files.

Terragrunt analyzes these dependencies to determine the correct order of operations:

1. Discovery: Terragrunt discovers configurations that might be relevant to a run based on the current working directory.
2. Constructing the Queue: Based on the command being run, Terragrunt creates an ordered queue.
   • For commands like plan or apply, dependencies are run before the units that depend on them.
   • For commands like destroy, dependent units are run before their dependencies.
3. Runs: Terragrunt dequeues the units in the queue and runs them, respecting the queue order. By default, it runs units concurrently up to a certain limit (controlled by the --parallelism flag), but it will always wait for a unit’s dependencies (or dependents for destroys) to complete successfully before running that unit.

Consider a setup where:

• Unit “dependent” depends on unit “dependency”.
• Unit “dependency” depends on unit “ancestor-dependency”.
• Unit “independent” has no dependencies nor dependents.

Assuming a current working directory of the root directory, Terragrunt would run units in the following order:

• run --all plan Order: Terragrunt would run independent and ancestor-dependency concurrently. Once ancestor-dependency finishes, dependency would run. Once dependency finishes, dependent would run.
• run --all destroy Order: Terragrunt would run dependent and independent concurrently. Once dependent finishes, dependency would run. Once dependency finishes, ancestor-dependency would run.

To visualize the dependency graph you can use the dag graph command (similar to the tofu/terraform graph command), or its equivalent list --format=dot --dependencies --external command.

The graph is output in DOT format. The typical program used to render this file format is GraphViz, but many web services are available that can do this as well.

terragrunt dag graph | dot -Tsvg > graph.svg
# Or equivalently:
terragrunt list --format=dot --dependencies --external | dot -Tsvg > graph.svg

For a stack with frontend-app, backend-app, mysql, valkey, and vpc units (where frontend-app depends on backend-app and vpc, backend-app depends on mysql/valkey/vpc, and mysql/valkey depend on vpc), the rendered graph looks like:

The arrow points from a unit to its dependency (e.g. frontend-app depends on backend-app).

For plan and apply, Terragrunt runs units in the opposite direction of the arrow (e.g. backend-app is applied before frontend-app). For destroy (and plan/apply -destroy), Terragrunt runs in the direction of the arrow (e.g. frontend-app is destroyed before backend-app).

Several flags allow you to customize how Terragrunt builds and executes the run queue. By default, Terragrunt will include all units that are in the current working directory.

By default, when using the --all flag, Terragrunt will include all units that are in the current working directory.

Using any positive filter triggers “Exclude by default” behavior, meaning that Terragrunt will no longer automatically include all units in the current working directory, and will instead selectively include units only if they match a positive filter (and don’t match any negative filters).

More on this in the Filtering Units section.

You can control which units are included or excluded from the queue using the --filter flag.

Any filter that doesn’t start with a ! prefix is considered a positive filter. When Terragrunt sees that any positive filter is present, it will evaluate every path it encounters while walking the directory tree, and only include units that match a positive filter.

terragrunt run --all --filter './subtree/**' -- plan

This will tell Terragrunt only to run the units that match the glob pattern ./subtree/** (any directory found in the current working directory starting with subtree).

There are many more types of filters you can use, and those details are covered in the Filter feature documentation.

Any filter that starts with a ! prefix is considered a negative filter. Negative filters are always evaluated after positive filters (if present).

terragrunt run --all --filter '!./subtree/**' -- plan

This will tell Terragrunt to run all units in the current working directory except those that match the glob pattern ./subtree/** (any directory found in the current working directory starting with subtree).

You can combine positive and negative filters to ensure that only the units you want are run.

terragrunt run --all --filter './subtree/**' --filter '!./subtree/dependency/**' -- plan

This will tell Terragrunt to run all units in the directory subtree except those that match the glob pattern ./subtree/dependency/** (any directory found in the current working directory starting with subtree/dependency).

• --queue-construct-as (--as): Build the run queue as if a particular command was run. Useful for performing dry-runs of run using discovery commands, like find and list.

  e.g. terragrunt list --queue-construct-as destroy

  This lists the units in the order they’d be processed for run --all destroy:

  $ terragrunt list --as destroy -l
  Type  Path
  unit  independent
  unit  subtree/dependent
  unit  subtree/dependency
  unit  ancestor-dependency

  $ terragrunt list --as plan -l
  Type  Path
  unit  ancestor-dependency
  unit  independent
  unit  subtree/dependency
  unit  subtree/dependent

• --queue-ignore-dag-order: Run units in the queue concurrently without respecting the dependency order.

  e.g. terragrunt run --all plan --queue-ignore-dag-order

  Run plan on ancestor-dependency, subtree/dependency, subtree/dependent, and independent all concurrently, without waiting for their defined dependencies. For instance, subtree/dependent’s plan would not wait for subtree/dependency’s plan to complete.

• --queue-ignore-errors: Continue processing the queue even if some units fail.

  e.g. terragrunt run --all plan --queue-ignore-errors

  If ancestor-dependency’s plan fails, Terragrunt will still attempt to run plan for subtree/dependency, then subtree/dependent, and also for independent.

• --fail-fast: Fail the run if any unit fails, stopping all remaining units immediately.

  e.g. terragrunt run --all plan --fail-fast

  If independent’s plan fails, Terragrunt will stop running any more units and fail the run, even if ancestor-dependency’s plan succeeds.

By default, Terragrunt does not impose a limit on the number of units it executes when it traverses the dependency graph. If it finds 5 units without dependencies, it runs OpenTofu/Terraform 5 times in parallel, once in each unit.

This can create problems when a graph has many units, like hitting a cloud provider’s rate limit.

Use the --parallelism [number] flag to cap the maximum number of unit executions at any given time:

terragrunt run --all --parallelism 4 -- apply

terragrunt stack output reads unit outputs in parallel. Use the same --parallelism flag to lower concurrency if a large stack hits resource limits (for example, file descriptors or cloud-provider rate limits):

terragrunt stack output --parallelism 4

A powerful feature of OpenTofu/Terraform is the ability to save the result of a plan as a binary file using the -out flag.

Terragrunt provides special tooling in run --all execution to ensure that the saved plan for a run --all against a stack has a corresponding entry for each unit in the stack in a directory structure that mirrors the stack structure.

To save every plan generated when running a stack, use the --out-dir flag (or TG_OUT_DIR environment variable) as demonstrated below:

$ terragrunt run --all --out-dir /tmp/tfplan -- plan

This integration also exists for the apply command, where the generated plan will be used when performing the apply.

$ terragrunt run --all --out-dir /tmp/tfplan -- apply

For planning a destroy operation, use the following commands:

terragrunt run --all --out-dir /tmp/tfplan -- plan -destroy
terragrunt run --all --out-dir /tmp/tfplan -- apply

To save plans in json format use the --json-out-dir flag:

terragrunt run --all --json-out-dir /tmp/json -- plan

terragrunt run --all --out-dir /tmp/all --json-out-dir /tmp/all -- plan