Skip to content

Support multiple_outputs in @task.bash - #71808

Open
dheerajturaga wants to merge 2 commits into
apache:mainfrom
dheerajturaga:bash-decorator-multiple-outputs
Open

Support multiple_outputs in @task.bash#71808
dheerajturaga wants to merge 2 commits into
apache:mainfrom
dheerajturaga:bash-decorator-multiple-outputs

Conversation

@dheerajturaga

@dheerajturaga dheerajturaga commented Aug 19, 2026

Copy link
Copy Markdown
Member

BashOperator already unrolls a dict return value into one XCom per key —
multiple_outputs is a BaseOperator parameter and the Task SDK runner honours it
regardless of which operator produced the value. Pairing it with output_processor
works on BashOperator today:

BashOperator(
    task_id="emit",
    bash_command="""echo '{"rows": 42, "uri": "s3://bucket/out"}'""",
    multiple_outputs=True,
    output_processor=json.loads,
)
# pushes: rows=42, uri="s3://bucket/out", return_value={"rows": 42, "uri": ...}

The @task.bash decorator, however, warned that the option was unsupported and
silently forced it off, so Dag authors could not name individual outputs from a Bash
task even though the operator it wraps supported it. This drops the warning and lets
the value through, which also makes @task.bash consistent with the other decorators
in the provider (@task.virtualenv, @task.short_circuit, …), all of which pass
multiple_outputs through.

Docs gained a "Multiple XCom outputs" section — the existing "Output processor"
section never mentioned multiple_outputs, which made the capability easy to miss.
Both example Dags gained a producer/consumer task pair so the documented snippets come
from code that CI parses.

The docs are deliberately explicit about the constraints, since only the last line of
the merged stdout/stderr stream is captured: a trailing blank echo or a subcommand
writing to stderr last will overwrite the dict, the JSON must be single-line, and
every emitted line reaches the task log.

image
Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

The BashOperator already unrolls a dict return value into one XCom per key,
since multiple_outputs is a BaseOperator parameter honoured by the task
runner. The @task.bash decorator warned that the option was unsupported and
silently forced it off, so Dag authors could not name individual outputs from
a Bash task even though the underlying operator supported it.
Airflow 2.11 bundles its own `@task.bash` decorator that ignores
`multiple_outputs` and warns about it. The compatibility test suite imports
`task` from `airflow.decorators` on Airflow 2, so the test exercises that
bundled decorator rather than the provider's, and the new behaviour is not
reachable there.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant