Description
The Iceberg materialization connector submits Spark jobs to EMR Serverless but doesn't tag them with anything. If a customer runs multiple materializations through a single EMR application, there's no way to tell which jobs belong to which pipeline for cost breakdowns, filtering in the console, auditing etc.
Request
Add an estuary:materialization tag with the full task name (e.g. acmeCo/prod/materialize-iceberg) to every StartJobRun call. The value is already available as materializationName on the emrClient struct - this is a one-line addition to the StartJobRunInput in materialize-iceberg/emr.go.
The job Name field already contains the materialization name and is visible in the EMR console, but it can't be used for cost allocation or programmatic filtering. Only tags work for that.
IAM consideration
Passing tags on StartJobRun likely requires emr-serverless:TagResource permission. Our documented IAM policy doesn't include it today, so adding tags unconditionally would break existing customers whose policies only grant StartJobRun. Therefore, we should probably make the tag opt-in via spec/config.
Description
The Iceberg materialization connector submits Spark jobs to EMR Serverless but doesn't tag them with anything. If a customer runs multiple materializations through a single EMR application, there's no way to tell which jobs belong to which pipeline for cost breakdowns, filtering in the console, auditing etc.
Request
Add an
estuary:materializationtag with the full task name (e.g.acmeCo/prod/materialize-iceberg) to everyStartJobRuncall. The value is already available asmaterializationNameon theemrClientstruct - this is a one-line addition to the StartJobRunInput inmaterialize-iceberg/emr.go.The job
Namefield already contains the materialization name and is visible in the EMR console, but it can't be used for cost allocation or programmatic filtering. Only tags work for that.IAM consideration
Passing tags on
StartJobRunlikely requiresemr-serverless:TagResourcepermission. Our documented IAM policy doesn't include it today, so adding tags unconditionally would break existing customers whose policies only grantStartJobRun. Therefore, we should probably make the tag opt-in via spec/config.