11,976 questions
0
votes
0
answers
82
views
Local unit testing apache airflow
Cannot Run Apache Airflow Unit Tests: sqlalchemy.exc.OperationalError: unable to open database file
I am attempting to run a single unit test within the Apache Airflow repository (airflow-core/tests/...
- 21
1
vote
1
answer
46
views
XCom limit of 1024 in airflow
I am hitting the 1024 item limit in the size of a returned list in my task in airflow. I am currently on airflow v2.11. A simple bit of code that reproduces the issue I see is below.
import os, ...
0
votes
0
answers
43
views
What cause this Airflow EmailOperator Error?
I am using Airflow 3.x for a personal task and faced this error
Failure caused by [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1010) source=task.stdout
What I want to do is send an email ...
- 1
-1
votes
1
answer
53
views
How to trigger a downstream Airflow DAG only when specific tasks in another DAG succeed?
I’m using Apache Airflow 2.x and I want to trigger a downstream DAG only when specific tasks in an upstream DAG succeed.
Here’s the situation:
I have an upstream DAG: dag_A
Inside dag_A, I have ...
1
vote
1
answer
63
views
Depedency Hell airflow+dbt
Hello I'm new to airflow, lately I'm struggling on a project with dbt+airflow+docker. My problem 1) I pip install dbt-core, dbt-duckdb adapter, 2) I try to install airflow with:
pip install "...
- 11
0
votes
1
answer
51
views
Add jinja template value with DateTime value
In an Airflow Dag file, I'm trying to handle following jinja template variables as DateTime values.
Specifically {{ data_interval_start }} and {{ data_interval_end }},
which (according to https://...
- 1,434
0
votes
0
answers
42
views
What does "state attribute is queued" mean for airflow 3?
I try to install the new airflow 3 with docker on a VM with this conf:
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE}
restart: unless-stopped
env_file: .env
environment:
# --...
- 2,704
Best practices
0
votes
0
replies
44
views
How to manage (Google Cloud Composer) Airflow roles with infrastructure as code?
Are there any best practices for handling roles in code?
Most of our setup is managed via Terraform. It would be great if there was a possibility.
There is a Terraform provider, however using the ...
- 3,731
0
votes
0
answers
28
views
Great Expectations 1.8.0 – InvalidDataContextConfigError: “Error while processing DataContextConfig: anonymous_usage_statistics validations_store_name
I’m using Great Expectations 1.8.0 inside an Apache Airflow DAG (Python 3.12), and I keep getting this error whenever my task tries to initialize the FileDataContext:
InvalidDataContextConfigError: ...
0
votes
0
answers
50
views
KubernetesPodOperator - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
I’m running Apache Airflow inside a Docker container and trying to use the KubernetesPodOperator to run a simple “hello world” pod in an external Kubernetes cluster (not the same one where Airflow ...
Tooling
1
vote
0
replies
63
views
masking secrets airflow 2.4.3
I’m using Apache Airflow 2.4.3 and trying to securely store a Snowflake connection with a private key inside the connection’s extras JSON field.
I want to mask the sensitive private_key_content field ...
0
votes
0
answers
35
views
Airflow 3 - Run LocalExecutor with unlimited Parallelism
Airflow Docs say that the LocalExecutor supports an "unlimited Parallelism". Source
They do not specify exactly how to run LocalExecutor in this mode, but I assume it is through the core....
- 1,255
0
votes
0
answers
78
views
MWAA Airflow task_instance_mutation_hook not working
I have added a airflow_local_setting.py file to my mwaa s3 bucket at the root of dags folder. I have a dag_policy defined which is being picked up and works as expected.
In the same file I have added ...
- 13
0
votes
1
answer
123
views
Unable to use Airflow variable with function in Jinja template
I'm trying to pass Airflow logical_date to dbt model so that I can use it in the model (sql). I'm using Airflow 2.11.0. I'm doing below, but the dag couldn't get constructed, with an error says:
'...
- 367
0
votes
0
answers
201
views
How can I dynamically refresh a JWT token in Airflow with Microsoft SSO using RSA?
I’ve configured Microsoft SSO with Apache Airflow using RSA-based authentication. The setup involves uploading the public key to the Azure App Registration, while Airflow holds the private key to ...
- 19