Queues created by Shovels aren't quorum queues despite providing x-queue-type
#9951
-
|
RabbitMQ So I have a few hundred shovels, all of which will use an exchange for Here's an example of a shovel definition that I use. {
"parameters": [
{
"component": "shovel",
"name": "from-centralized-services",
"value": {
"ack-mode": "on-confirm",
"dest-add-forward-headers": true,
"dest-exchange": "event",
"dest-protocol": "amqp091",
"dest-uri": "amqp:///MDM5M3R4NTc4OHk4aHdkbmM4bnM2M2ljXzI3bndmZmFmZHU0eWszbXptOWwwanBtdHkK",
"src-delete-after": "never",
"src-exchange": "event",
"src-exchange-key": "*.anonymized123.annoymized123.#",
"src-prefetch-count": 1,
"src-protocol": "amqp091",
"src-uri": "amqp:///central-services",
"dest-queue-args": {
"x-queue-type": "quorum"
}
},
"vhost": "MDM5M3R4NTc4OHk4aHdkbmM4bnM2M2ljXzI3bndmZmFmZHU0eWszbXptOWwwanBtdHkK"
}
]
}And the prerequisite definitions for above shovel to work. {
"vhosts": [
{
"limits": [
["max-queues", 150],
["max-connections", 10]
],
"metadata": {
"description": "Empty",
"tags": ["Empty"]
},
"name": "MDM5M3R4NTc4OHk4aHdkbmM4bnM2M2ljXzI3bndmZmFmZHU0eWszbXptOWwwanBtdHkK"
}
],
"bindings": [
{
"arguments": {},
"destination": "amq.gen-G7wpNgJ0OzMYrBlMfeDeYg",
"destination_type": "queue",
"routing_key": "*.anonymized123.anonymized123.#",
"source": "command",
"vhost": "MDM5M3R4NTc4OHk4aHdkbmM4bnM2M2ljXzI3bndmZmFmZHU0eWszbXptOWwwanBtdHkK"
}
],
"exchanges": [{
"arguments": {},
"auto_delete": false,
"durable": true,
"name": "event",
"type": "topic",
"vhost": "MDM5M3R4NTc4OHk4aHdkbmM4bnM2M2ljXzI3bndmZmFmZHU0eWszbXptOWwwanBtdHkK"
}]
}The above shovel ends up creating a classic queue prefixed with Even if I set up the below policy to force mirroring classic queues, it only works for classic queues not created by shovels. rabbitmqctl set_policy ha-exactly-two ".*" '{"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"}' -p "MDM5M3R4NTc4OHk4aHdkbmM4bnM2M2ljXzI3bndmZmFmZHU0eWszbXptOWwwanBtdHkK"This is the guide that I followed. Any tips/suggestions are highly appreciated. Thank you! |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 11 replies
-
|
RabbitMQ 3.9 is completely out of support. Classic mirrored queues have been deprecated for years (see at the very top) and are scheduled to be removed very soon (#9815). |
Beta Was this translation helpful? Give feedback.
-
|
Yeah I also don't want to use classic queues but by default the shovels would create a classic queue if we define an exchange as the destination. |
Beta Was this translation helpful? Give feedback.
-
|
Exclusive queues cannot be mirrored by definition: their lifecycle is tied to that of a single connection (to a specific node). You can override queue name and arguments for Dynamic shovels. |
Beta Was this translation helpful? Give feedback.
-
|
Given this is not a fix, but a change in behaviour that has been there forever, I'm not sure it'll back ported at all, but if it is, then yes - certainly not to versions older than 3.11, since they are unsupported already. The solution you can use right now is to switch to Please don't use outdated/unsupported software versions. |
Beta Was this translation helpful? Give feedback.
-
|
@mkuratczyk ah, OK. We discussed this already and there are no objections to changing this behavior for 3.13. You can always set up any topology you need using e.g. a definitions import and make Shovels use pre-existing exchanges. |
Beta Was this translation helpful? Give feedback.
-
|
Appreciate the insights @mkuratczyk and yes I am planning to migrate this cluster to 3.12 without a downtime and this issue/behavior kinda got in the way of this whole "no downtime" plan I had. I will check if we can define the src-queue instead and take it from there. Thanks guys! |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin @mkuratczyk, Apologies for re-asking something that seems to have been already answered but can you confirm if you guys have decided to provide the following behavior of the shovel plugin with the release of 3.13
|
Beta Was this translation helpful? Give feedback.
-
|
We have decided to do this, but no promises it will make it into 3.13, strictly due to time constraints. A PR could be accepted quickly |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin @mkuratczyk |
Beta Was this translation helpful? Give feedback.
-
|
Two things:
|
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately not, the RMQ we are using is provided by our Private Cloud provider, we do not control it. We are rather just users who can create their own stuff within existing/provided clusters. That's the reason I'm asking theoretical questions instead of testing it myself 😄Basically, I'm trying to prepare myself and our setup before the migration takes place. I'm going to reach out the vendor. However, asking at the source is rather better in my opinion. |
Beta Was this translation helpful? Give feedback.
-
|
Your shovels should "just work" after an upgrade. Since we don't know how they are defined, that's about all I can say. Classic queues exist and are supported in RabbitMQ 4, you just can't mirror them. PS it's very easy to run RabbitMQ locally or via a docker container to try things out yourself. |
Beta Was this translation helpful? Give feedback.
amq.gen-means a server-named, very likely exclusive queue.Exclusive queues cannot be mirrored by definition: their lifecycle is tied to that of a single connection (to a specific node).
You can override queue name and arguments for Dynamic shovels.