Bulk Activity extract - Create job hangs

[smannam07]

marketorestpython api to get bulk activities data in databricks is just hanging without any error message.

new_export_job_details = mc.execute(method='create_activities_export_job', fields=['string', 'string'], filters={'createdAt': {'endAt': '2017-11-02', 'startAt': '2017-11-01'}})

Please advise.

[jepcastelein]

If you used fields=['string', 'string'] I would first remove that, those are not valid field names and fields is optional anyways. If that doesn't do the trick, I would try it in your Terminal first.

[smannam07]

I just copied the call in the question. In my code, I am not using the fields. Here is what I am using.

new_export_job_details = mc.execute(method='create_activities_export_job', filters={'createdAt': {'endAt': '2021-07-12', 'startAt': '2021-07-10'}})

How to try in Terminal?

[jepcastelein]

what I mean is if you have Python installed on your local machine, to just open the command line or terminal, start python, import marketorestpython, instantiate it and make the call. Then you should get any error messages directly thrown into the command line. I am not familiar with Databricks and how it handles errors, so it's better to try it outside of such a tool. You could also use a Python IDE. I just did this for my own Marketo instance and used your exact code and it works, response is:

[{'exportId': '029d5799-56b5-4024-b226-51b349c2af2c', 'format': 'CSV', 'status': 'Created', 'createdAt': '2021-07-14T01:35:50Z'}]

[java8964]

yes, I can reproduce this issue on version marketorestpython==0.5.14

I think the hanging happens when the following error happened -> "Marketo API Error Code 1029: Export daily quota 500MB exceeded."

Here is the stack trace:
Traceback (most recent call last):
File "/Users/xxx/venv/python3.7/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry
ret = target(*args, **kwargs)
File "/Users/xxx/venv/python3.7/lib/python3.7/site-packages/marketorestpython/helper/http_lib.py", line 50, in rateLimitedFunction
ret = func(*args,**kargs)
File "/Users/xxx/venv/python3.7/lib/python3.7/site-packages/marketorestpython/helper/http_lib.py", line 92, in post
raise MarketoException(r_json['errors'][0])
marketorestpython.helper.exceptions.MarketoException: Marketo API Error Code 1029: Export daily quota 500MB exceeded.

I am not very familiar with the Python, but it looks like for this particular error, the code went to retry_exception method in _sync.py, and just stuck here forever.

[java8964]

I think I understand more about this issue as now.

First, the error code 1029 returned by Marketo could mean 2 things, based on the document https://developers.marketo.com/rest-api/error-codes/

• Too many jobs in queue: Subscriptions are allowed a maximum of 10 bulk extract jobs in the queue at any given time.
• Export daily quota exceeded: By default extract jobs are limited to 500MB per day (resets daily at 12:00AM CST).

The current code (Or as version 0.5.14) treats error code 1029 as recoverable -> https://github.com/jepcastelein/marketo-rest-python/blob/master/marketorestpython/helper/http_lib.py#L20

This maybe makes sense for the 1st situation, but definitely makes the code look like hanging for the 2nd case, as it is NOT recoverable up to 24 hours. We may want to remove the 1029 from the list of retryable_error_codes, so the code should fail fast for such case.

PR -> #69

[jepcastelein]

thanks for identifying this issue! I think I want to leave the "Too many jobs in queue" as retryable, but indeed the "daily quota exceeded" should fail right away. I will change https://github.com/jepcastelein/marketo-rest-python/blob/master/marketorestpython/helper/http_lib.py#L26 to:

return e.code not in retryable_error_codes or 'Export daily quota' in e.message