[Questions] There are quorum queues that have lost a node online

[kamilbro5]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.1.1

Erlang version used

27.3.x

Operating system (distribution) used

Red Hat Enterprise Linux 9.6

How is RabbitMQ deployed?

RPM package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

N/A

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

2025-12-06 07:47:38.221451+01:00 [info] <0.2086.0> queue 'PushService_SessionUpdated' in vhost '/': Leader monitor down with {{bad_return_from_state_function,{error,[100,101|...]}},[{gen_statem,loop_state_callback_result,11,[...]},{proc_lib,init_p_do_apply,3,...}]}, setting election timeout
2025-12-06 07:47:38.257942+01:00 [info] <0.2086.0> queue 'PushService_SessionUpdated' in vhost '/': granting vote for {'%2F_PushService_SessionUpdated',node2@s2slp0016675} with last {index, term} {32090369,50} for term 51 previous term was 51
2025-12-06 07:47:38.263630+01:00 [info] <0.2086.0> queue 'PushService_SessionUpdated' in vhost '/': detected a new leader {'%2F_PushService_SessionUpdated',node2@s2slp0016675} in term 51

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

2025-12-06 07:47:38.221490+01:00 [info] <0.1947.0> queue 'PushService_SessionUpdated' in vhost '/': Leader monitor down with {{bad_return_from_state_function,{error,[100,101|...]}},[{gen_statem,loop_state_callback_result,11,[...]},{proc_lib,init_p_do_apply,3,...}]}, setting election timeout
2025-12-06 07:47:38.262670+01:00 [notice] <0.1947.0> queue 'PushService_SessionUpdated' in vhost '/': candidate -> leader in term: 51 machine version: 6, last applied 32090369
2025-12-06 08:42:00.702275+01:00 [info] <0.1903.0> queue 'Snt_Reporting_TransactionsProcessed' in vhost '/': Leader monitor down with {{bad_return_from_state_function,{error,[100,101|...]}},[{gen_statem,loop_state_callback_result,11,[...]},{proc_lib,init_p_do_apply,3,...}]}, setting election timeout
2025-12-06 08:42:00.769506+01:00 [notice] <0.1903.0> queue 'Snt_Reporting_TransactionsProcessed' in vhost '/': candidate -> leader in term: 26 machine version: 6, last applied 4413268

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

2025-12-01 11:00:27.442055+01:00 [info] <0.1777.0> queue 'Session_AspspCallMade' in vhost '/': Leader monitor down with {{bad_return_from_state_function,{error,[100,101|...]}},[{gen_statem,loop_state_callback_result,11,[...]},{proc_lib,init_p_do_apply,3,...}]}, setting election timeout
2025-12-01 11:00:27.532275+01:00 [notice] <0.1777.0> queue 'Session_AspspCallMade' in vhost '/': candidate -> leader in term: 22 machine version: 6, last applied 29681456
2025-12-01 17:39:56.737181+01:00 [info] <0.1714.0> queue 'PushService_SessionUpdated' in vhost '/': Leader monitor down with {{bad_return_from_state_function,{error,[100,101|...]}},[{gen_statem,loop_state_callback_result,11,[...]},{proc_lib,init_p_do_apply,3,...}]}, setting election timeout
2025-12-01 17:39:56.825635+01:00 [notice] <0.1714.0> queue 'PushService_SessionUpdated' in vhost '/': candidate -> leader in term: 49 machine version: 6, last applied 31130703
2025-12-01 17:58:09.216828+01:00 [info] <0.1636.0> queue 'PushService_ReportingSyncAllFinished' in vhost '/': Leader monitor down with {{bad_return_from_state_function,{error,[100,101|...]}},[{gen_statem,loop_state_callback_result,11,[...]},{proc_lib,init_p_do_apply,3,...}]}, setting election timeout
2025-12-01 17:58:09.267285+01:00 [info] <0.1636.0> queue 'PushService_ReportingSyncAllFinished' in vhost '/': granting vote for {'%2F_PushService_ReportingSyncAllFinished',node2@s2slp0016675} with last {index, term} {5491830,22} for term 23 previous term was 22
2025-12-01 17:58:09.275847+01:00 [info] <0.1636.0> queue 'PushService_ReportingSyncAllFinished' in vhost '/': detected a new leader {'%2F_PushService_ReportingSyncAllFinished',node2@s2slp0016675} in term 23

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

default_vhost = ZT002_api_rmq


# Log Settings
log.dir = /app/logs/rmq/node1
log.file = mylocalhostname
log.file.level = debug
log.file.rotation.date = $D0
log.file.rotation.count = 120
listeners.tcp.default = 4443
log.connection.level = info
log.channel.level = info
log.queue.level = info
log.mirroring.level = info
log.federation.level = info
log.upgrade.level = info
log.default.level = info

# SSL Settings
listeners.ssl.default = 9443
ssl_options.certfile = /etc/szef/requested_certs/mylocalhostname.crt
ssl_options.keyfile = /etc/szef/requested_certs/mylocalhostname.key
ssl_options.fail_if_no_peer_cert = false
ssl_options.verify = verify_none
ssl_options.versions.1 = tlsv1.2
ssl_options.honor_cipher_order = true
ssl_options.honor_ecc_order = true
ssl_options.ciphers.1 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.2 = DHE-RSA-AES128-SHA256
ssl_options.ciphers.3 = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.4 = DHE-RSA-AES256-SHA256
ssl_options.ciphers.5 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.6 = ECDHE-RSA-AES128-SHA256
ssl_options.ciphers.7 = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.8 = ECDHE-RSA-AES256-SHA384

# MGMT Settings
#management.tcp.port = 9060
management.ssl.port = 8443
management.ssl.idle_timeout =  120000
management.ssl.inactivity_timeout =  120000
management.ssl.request_timeout = 10000
management.ssl.certfile = /etc/szef/requested_certs/mylocalhostname.crt
management.ssl.keyfile = /etc/szef/requested_certs/mylocalhostname.key
management.ssl.versions.1 = tlsv1.2
management.ssl.honor_cipher_order = true
management.ssl.honor_ecc_order = true
management.ssl.client_renegotiation = false
management.ssl.secure_renegotiate = true
management.csp.policy = default-src 'self' *.mylocalhostname 'unsafe-eval' 'unsafe-inline' frame-ancestors 'none'
management.hsts.policy = max-age=31536000; includeSubDomains
management.cors.allow_origins.1 = *.mylocalhostname
management.cors.allow_origins.2 = *.mylocalhostname
management.cors.max_age = 3600
management.headers.content_type_options = nosniff
management.headers.xss_protection = 1; mode=block
management.headers.frame_options = SAMEORIGIN
management.ssl.ciphers.1 = DHE-RSA-AES128-GCM-SHA256
management.ssl.ciphers.2 = DHE-RSA-AES128-SHA256
management.ssl.ciphers.3 = DHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.4 = DHE-RSA-AES256-SHA256
management.ssl.ciphers.5 = ECDHE-RSA-AES128-GCM-SHA256
management.ssl.ciphers.6 = ECDHE-RSA-AES128-SHA256
management.ssl.ciphers.7 = ECDHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.8 = ECDHE-RSA-AES256-SHA384
management.sample_retention_policies.global.minute  = 5
management.sample_retention_policies.global.hour    = 60
management.sample_retention_policies.global.day     = 1200
management.sample_retention_policies.basic.minute 	= 5
management.sample_retention_policies.basic.hour   	= 60
management.sample_retention_policies.detailed.10 	= 5

default_user_tags.administrator = true
default_user_tags.management = true

# DISK limits
disk_free_limit.relative = 0.80

# MEM Settings
vm_memory_calculation_strategy = rss
vm_memory_high_watermark.relative = 0.87
vm_memory_high_watermark_paging_ratio = 0.75

# LDAP Settings
auth_backends.1 = internal
auth_backends.2 = cache
auth_cache.cached_backend = ldap
auth_cache.cache_ttl = 300000
auth_ldap.servers.1  = ldap2012.mylocalhostname
auth_ldap.user_dn_pattern = CN=${username},CN=A,DC=A,DC=A
auth_ldap.use_ssl    = true
auth_ldap.port       = 636
auth_ldap.log        = false
auth_ldap.timeout    = 30000
auth_ldap.idle_timeout = 18000


# Quorum queue properties
quorum_queue.continuous_membership_reconciliation.enabled = true
quorum_queue.continuous_membership_reconciliation.target_group_size = 3
quorum_queue.continuous_membership_reconciliation.auto_remove = true

# Custom config from databags
disk_free_limit.relative = 0.80
cluster_partition_handling  = autoheal
message_interceptors.incoming.set_header_timestamp.overwrite = false

Steps to deploy RabbitMQ cluster

yum install rabbitmq

Steps to reproduce the behavior in question

My cluster behaves correctly after startup.
After connecting clients, online nodes drop out of the queues at random times.
A dotNET/.NET client is connected to the queues.
When an online node drops out of the queue (it's always a random node), there are no entries in the main log, connection log, or queue.log

I see a small peak of messages when a node drops out of the queue, about 30 messages at a time, I monitor it

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

[
  {rabbit, [
      {config_entry_decoder, [
             {passphrase,{ file, "/app/rmq/node1/config_pass"} },
             {cipher, blowfish_cfb64},
             {hash, sha256},
             {iterations, 10000}
         ]},
      {credential_validator, [{validation_backend, rabbit_credential_validator_password_regexp},
                                   {regexp, <<"^[a-zA-Z0-9\S$]{2,100}">>}]
        },
      {default_user, admin},
      {default_pass,
             {encrypted, <<"AAAAAAAAAAAAAAAAAAAAAA">> }}
    ]},
   {rabbitmq_auth_backend_ldap,[
     {group_lookup_base, "DC=A,DC=A"},
     {ssl_options, [
                    {cacertfile,"/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"},
                    {verify, verify_peer},
                    {fail_if_no_peer_cert, false}

     ]},
     {vhost_access_query, {'or', [{in_group, "CN=A,OU=Distribution,OU=Groups,OU=_Exchange,DC=A,DC=A"},
                          {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=Groups,OU=A,DC=A,DC=A"},
                          {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=Groups,OU=A,DC=A,DC=A"}]} },
     {tag_queries, [{administrator, {in_group, "CN=A,OU=Distribution,OU=Groups,OU=_Exchange,DC=A,DC=A"}},
                    {monitoring, {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=Groups,OU=A,DC=A,DC=A"}},
                    {management, {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=Groups,OU=A,DC=A,DC=A"}}
                    ] }    ]},
    {lager, [
        {error_logger_hwm, 1024}
    ]}
].

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

rabbitmq-queues delete_member -p / Crm_TransactionMessage node2@hostname
rabbitmq-queues add_member -p / Crm_TransactionMessage node2@hostname
or
restart nodes...
after 1-3 days production operation still node leave queueus

[michaelklishin]

We cannot guess what happens to your node. We are engineers, not magicians.

See node logs on every node for clues.

[kamilbro5]

I attached all logs from nodes which i found

[kjnilsson]

Did you note any errors or anything else of interest?

[michaelklishin]

@kamilbro5 you have attached mere 14 lines in total to the questionnaire.

[michaelklishin]

@kjnilsson see the expandable log sections in the issue description. There is a number of leader elections triggered by a bad_return_from_state_function error:

2025-12-01 11:00:27.442055+01:00 [info] <0.1777.0> queue 'Session_AspspCallMade' in vhost '/': Leader monitor down with {{bad_return_from_state_function,{error,[100,101|...]}},[{gen_statem,loop_state_callback_result,11,[...]},{proc_lib,init_p_do_apply,3,...}]}, setting election timeout
2025-12-01 11:00:27.532275+01:00 [notice] <0.1777.0> queue 'Session_AspspCallMade' in vhost '/': candidate -> leader in term: 22 machine version: 6, last applied 29681456
2025-12-01 17:39:56.737181+01:00 [info] <0.1714.0> queue 'PushService_SessionUpdated' in vhost '/': Leader monitor down with {{bad_return_from_state_function,{error,[100,101|...]}},[{gen_statem,loop_state_callback_result,11,[...]},{proc_lib,init_p_do_apply,3,...}]}, setting election timeout

[kjnilsson]

It does look like there could be a potential issue worth investigating but there should be more crash details available.

[kamilbro5]

I was looking in all logs files and /var/log/messeages and i didn't found nothig more

[michaelklishin]

RabbitMQ nodes log a fair amount at info and higher levels, so having just those entries in the log would be a truly unique situation like log rotation resetting the current log file shortly before the event.

Again, it's on you to understand how the logging is configured and where to get the logs. Maybe your nodes log to standard streams now, something very typical with containerized deployments.

[kamilbro5]

maybe should i set logs level to debug?

[michaelklishin]

That's a standard procedure when investigating something.

[kamilbro5]

Ok, i enabled debug level and restarted cluster yeasterday at 5 p.m. and i catch exacly my problem in logs.
At 9:28 p.m. zabbix monitoring catch remove node1 from online nodes on queue PolishApiConnector_Bank_1_auth

this is debug logs:

2025-12-08 21:28:39.823693+01:00 [info] <0.2350.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': Leader monitor down with {{bad_return_from_state_function,{error,[100,
101|...]}},[{gen_statem,loop_state_callback_result,11,[...]},{proc_lib,init_p_do_apply,3,...}]}, setting election timeout
2025-12-08 21:28:39.870858+01:00 [debug] <0.2350.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': pre_vote election called for in term 30
2025-12-08 21:28:39.873313+01:00 [debug] <0.2350.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': follower -> pre_vote in term: 30 machine version: 6, last applied 193
8118
2025-12-08 21:28:39.873474+01:00 [debug] <0.2350.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': pre_vote granted #Ref<0.2912825774.2045771778.200883> for term 30 vot
es 1
2025-12-08 21:28:39.874066+01:00 [debug] <0.2350.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': pre_vote granted #Ref<0.2912825774.2045771778.200883> for term 30 vot
es 2
2025-12-08 21:28:39.874145+01:00 [debug] <0.2350.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': election called for in term 31
2025-12-08 21:28:39.876058+01:00 [debug] <0.2350.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': pre_vote -> candidate in term: 31 machine version: 6, last applied 19
38118
2025-12-08 21:28:39.876161+01:00 [debug] <0.2350.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': vote granted for term 31 votes 1
2025-12-08 21:28:39.888314+01:00 [debug] <0.2350.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': vote granted for term 31 votes 2
2025-12-08 21:28:39.888524+01:00 [notice] <0.2350.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': candidate -> leader in term: 31 machine version: 6, last applied 193
8118
2025-12-08 21:28:39.889116+01:00 [debug] <0.23612.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': rabbit_fifo_client Detected QQ leader change from {'%2F_PolishApiCon
nector_PkoBP_1_auth',node1@s1slp0016674} to {'%2F_PolishApiConnector_Bank_1_auth',node2@s2slp0016675}, resending 0 pending commands
2025-12-08 21:28:39.889167+01:00 [debug] <0.24724.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': rabbit_fifo_client Detected QQ leader change from {'%2F_PolishApiCon
nector_PkoBP_1_auth',node1@s1slp0016674} to {'%2F_PolishApiConnector_Bank_1_auth',node2@s2slp0016675}, resending 0 pending commands
2025-12-08 21:28:39.889182+01:00 [debug] <0.24248.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': rabbit_fifo_client Detected QQ leader change from {'%2F_PolishApiCon
nector_PkoBP_1_auth',node1@s1slp0016674} to {'%2F_PolishApiConnector_Bank_1_auth',node2@s2slp0016675}, resending 0 pending commands
2025-12-08 21:28:39.889341+01:00 [debug] <0.26040.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': rabbit_fifo_client Detected QQ leader change from {'%2F_PolishApiCon
nector_PkoBP_1_auth',node1@s1slp0016674} to {'%2F_PolishApiConnector_Bank_1_auth',node2@s2slp0016675}, resending 0 pending commands
2025-12-08 21:28:39.889452+01:00 [debug] <0.1438872.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': updating leader record to current node node2@s2slp0016675
2025-12-08 21:28:39.889373+01:00 [debug] <0.27590.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': rabbit_fifo_client Detected QQ leader change from {'%2F_PolishApiCon
nector_PkoBP_1_auth',node1@s1slp0016674} to {'%2F_PolishApiConnector_Bank_1_auth',node2@s2slp0016675}, resending 0 pending commands
2025-12-08 21:28:39.892787+01:00 [debug] <0.2350.0> queue 'PolishApiConnector_Bank_1_auth' in vhost '/': enabling ra cluster changes in 31, index 1938119

[kjnilsson]

can I have the logs from all nodes at the same time window please. the first log message would have a more comprehensive log on one of the other nodes.

[michaelklishin]

Note that this is on 4.1.1, which is almost six releases behind in the 4.1.x series alone.

[kjnilsson]

Not that much has changed in terms of QQ / Ra so details would still be valuable.

[kamilbro5]

sorry, this is my prod environment, i will try restart and enable debug log today in the evening

[kjnilsson]

logs without debug from the relevant time window (covering restart) would probably be enough for me to investigate a bit further.

[kamilbro5]

this is quorum status for this queue, more logs from all nodes i attach in a few hours.

rabbitmq-queues quorum_status PolishApiConnector_Bank_1_auth:
Status of quorum queue PolishApiConnector_Bank_1_auth on node node2@s2slp0016675 ...
┌────────────────────┬────────────┬────────────┬────────────────┬──────────────┬──────────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name          │ Raft State │ Membership │ Last Log Index │ Last Written │ Last Applied │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├────────────────────┼────────────┼────────────┼────────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ node1@s1slp0016674 │ noproc     │ unknown    │ 1938118        │ 1938118      │ 1938118      │ 1938118      │ 1938082        │ 30   │ 6               │
├────────────────────┼────────────┼────────────┼────────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ node2@s2slp0016675 │ leader     │ voter      │ 1944913        │ 1944913      │ 1944913      │ 1944913      │ 1938118        │ 31   │ 6               │
├────────────────────┼────────────┼────────────┼────────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ node3@s0slp0016676 │ follower   │ voter      │ 1944913        │ 1944913      │ 1944913      │ 1944913      │ 1938118        │ 31   │ 6               │
└────────────────────┴────────────┴────────────┴────────────────┴──────────────┴──────────────┴──────────────┴────────────────┴──────┴─────────────────┘

[michaelklishin]

noproc means that the replica had terminated with an exception. We cannot why exactly but we do know that there were relevant changes since 4.1.1 and we are aware of only a couple of outstanding issues that can manifest themselves this way.

Start with upgrading to 4.2.1. 4.1.1 is seven releases behind and you are missing out on file handle leaks and directly relevant changes if there are thousands of quorum queues.