Librabbitmq 2.0.0 with Python 3 gives TypeError: can't pickle memoryview objects

[sam-ghosh]

I am using the latest master branch of the git repo https://github.com/celery/librabbitmq and installing librabbitmq==2.0.0 for Python 3.6 by following the instructions in the readme

Using the development version

You can clone the repository by doing the following:

$ git clone git://github.com/celery/librabbitmq.git

Then install it by doing the following:

$ cd librabbitmq
$ make install        # or make develop

This works fine (after installing certain binaries for c compliation in the OS), but when I then make a small a+b add task and call it with add.delay(2,2) it fails with the following error. I looked up and saw that Celery 4 uses json as serializer, so clearly it is not because if pickle serialization

1. Changing from librabbitmq to pyamqp broker works normally
2. Same exact situation in both MacOS and Ubuntu 16

[2018-04-30 23:40:02,956: CRITICAL/MainProcess] Unrecoverable error: SystemError('<built-in method _basic_recv of Connection object at 0x1099ec5e8> returned a result with an error set',)
Traceback (most recent call last):
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/messaging.py", line 624, in _receive_callback
    return on_m(message) if on_m else self.receive(decoded, message)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 570, in on_task_received
    callbacks,
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/strategy.py", line 145, in task_message_handler
    handle(req)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/worker.py", line 221, in _process_task_sem
    return self._quick_acquire(self._process_task, req)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/async/semaphore.py", line 62, in acquire
    callback(*partial_args, **partial_kwargs)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/worker.py", line 226, in _process_task
    req.execute_using_pool(self.pool)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/request.py", line 531, in execute_using_pool
    correlation_id=task_id,
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/concurrency/base.py", line 155, in apply_async
    **options)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/billiard/pool.py", line 1486, in apply_async
    self._quick_put((TASK, (result._job, None, func, args, kwds)))
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 813, in send_job
    body = dumps(tup, protocol=protocol)
TypeError: can't pickle memoryview objects

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/bootsteps.py", line 370, in start
    return self.obj.start()
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 320, in start
    blueprint.start(self)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 596, in start
    c.loop(*c.loop_args())
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/loops.py", line 88, in asynloop
    next(loop)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/async/hub.py", line 354, in create_loop
    cb(*cbargs)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/transport/base.py", line 236, in on_readable
    reader(loop)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/transport/base.py", line 218, in _read
    drain_events(timeout=0)
  File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/librabbitmq-2.0.0-py3.6-macosx-10.6-intel.egg/librabbitmq/__init__.py", line 227, in drain_events
    self._basic_recv(timeout)
SystemError: <built-in method _basic_recv of Connection object at 0x1099ec5e8> returned a result with an error set

[vgoklani]

Hey - can you please explain this step: "This works fine (after installing certain binaries for c compliation in the OS)"

What did you install and how did you do it? Thanks!

[hhstore]

ref:

• python3.6 can install py-amqp, it's ok for me.

This is a fork of amqplib which was originally written by Barry Pederson.
It is maintained by the Celery project, and used by kombu as a pure python alternative when librabbitmq is not available.
This library should be API compatible with librabbitmq.

• https://github.com/celery/py-amqp
• version 2 on pypi #118

[auvipy]

this library is mostly incomplete and less maintained. py-amqp is the recommended one

[marvin-hao]

'solo' task pool works well. 'prefork' gives pickle issues. 'gevent' and 'eventlet' hang forever.

[shanavas786]

task.delay also causes the same error

[spumer]

Any updates?