Airflow Xcom Exclusive -

dag = xcom_exclusive_pipeline()

@task def exclusive_push(): with r.lock("xcom:my_key", timeout=10): r.set("xcom:my_key", "my_value")

By default, operators like BashOperator or custom enterprise hooks might push output to XCom automatically if do_xcom_push is set to True. If you run thousands of tasks an hour, this creates immense database noise. airflow xcom exclusive

# Task A and Task B run in parallel task_a >> task_c task_b >> task_c

Unindexed or overloaded XCom reads/writes from highly parallelized dynamic tasks. from airflow

from airflow.decorators import dag, task from datetime import datetime import pandas as pd @dag(start_date=datetime(2026, 1, 1), schedule=None, catchup=False) def enterprise_data_pipeline(): @task def extract_user_demographics(): # Representing data extraction raw_data = "user_id": [101, 102], "country": ["US", "KR"] # If Custom Backend is active, this Dict/DataFrame securely saves to S3 return raw_data @task def process_demographics(demographics): # Airflow automatically resolves the XCom backend URI back into the raw object df = pd.DataFrame(demographics) processed_data = df.to_dict(orient="records") return processed_data # Setting up dependency seamlessly via Python function invocation user_data = extract_user_demographics() process_demographics(user_data) enterprise_data_pipeline() Use code with caution. Mixing TaskFlow with Traditional Operators

from airflow import DAG from airflow.operators.python import PythonOperator from datetime import datetime Crucial XCom Best Practices & Constraints The Custom

You need to tell Airflow to use the new backend class.

# Pushing context['ti'].xcom_push(key='my_key', value='my_value') # Pulling value = context['ti'].xcom_pull(task_ids='push_task', key='my_key') Use code with caution. Crucial XCom Best Practices & Constraints

The Custom Backend automatically saves the dataframe as a file in S3.