Description
Currently the EmrStepSensor only looks at one step at a time. Enhancing the EmrStepSensor to retrieve all steps on a specified cluster if no stepId was provided.
Use case / motivation
If no step_id was provided the EmrStepSensor wil retrieve all steps on a specified cluster.
This allows you to retrieve all steps and their states with a singe api call.
What happened
We ran into a throttle limit.
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeStep operation (reached max retries: 4): Rate exceeded
With the enhancement of the EmrStepSensor that is taken care of and also then the EmrStepSensor is more dynamic.
Description
Currently the EmrStepSensor only looks at one step at a time. Enhancing the EmrStepSensor to retrieve all steps on a specified cluster if no stepId was provided.
Use case / motivation
If no step_id was provided the EmrStepSensor wil retrieve all steps on a specified cluster.
This allows you to retrieve all steps and their states with a singe api call.
What happened
We ran into a throttle limit.
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeStep operation (reached max retries: 4): Rate exceeded
With the enhancement of the EmrStepSensor that is taken care of and also then the EmrStepSensor is more dynamic.