How to create RabbitMQ cluster

Create RabbitMQ cluster to replicate configuration across multiple nodes.

Preliminary information

I will create three node setup using nodes: rabbit (192.168.50.201), reindeer (192.168.50.202) and raccoon (192.168.50.203).

RabbitMQ uses a cookie to determine whether nodes can talk to each other. I will use string JFKZVCBYEISEQILVZMSD. It is an alphanumeric string up to 255 characters. Keep it secret.

Install RabbitMQ server on every node

Install RabbitMQ message broker on every node.

$ sudo apt update
$ sudo apt install gnupg2 apt-transport-https curl

$ curl https://dl.bintray.com/rabbitmq/Keys/rabbitmq-release-signing-key.asc | sudo apt-key add -

$ echo "deb https://dl.bintray.com/rabbitmq/debian stretch main"              | sudo tee    /etc/apt/sources.list.d/rabbitmq.list
$ echo "deb https://dl.bintray.com/rabbitmq-erlang/debian buster erlang-22.x" | sudo tee -a /etc/apt/sources.list.d/rabbitmq.list 

$ apt update
$ apt install rabbitmq-server

Use rabbitmq-diagnostics to display RabbitMQ version.

$ sudo rabbitmq-diagnostics server_version
Asking node [email protected] for its RabbitMQ version...
3.7.18

Setup first RabbitMQ node

Setup first RabbitMQ node - rabbit.

Define Erlang cookie.

$ sudo systemctl stop rabbitmq-server
$ echo "JFKZVCBYEISEQILVZMSD" | sudo tee /var/lib/rabbitmq/.erlang.cookie
$ sudo systemctl start rabbitmq-server

Enable management plugin.

$ rabbitmq-plugins enable rabbitmq_management

Define admin user with password password.

.
$ sudo rabbitmqctl add_user admin password
$ sudo rabbitmqctl set_user_tags admin administrator
$ sudo rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"

Delete guest user.

$ sudo rabbitmqctl delete_user guest

Setup additional RabbitMQ nodes

Setup additional RabbitMQ nodes - reindeer and raccoon.

Define Erlang cookie.

$ sudo systemctl stop rabbitmq-server
$ echo "JFKZVCBYEISEQILVZMSD" | sudo tee /var/lib/rabbitmq/.erlang.cookie
$ sudo systemctl start rabbitmq-server

Enable management plugin.

$ rabbitmq-plugins enable rabbitmq_management

Join cluster by specifying existing member.

$ sudo rabbitmqctl stop_app
$ sudo rabbitmqctl join_cluster [email protected]
$ sudo rabbitmqctl start_app

Verify cluster status

Verify cluster status.

$ sudo rabbitmqctl cluster_status
Cluster status of node [email protected] ...
Basics

Cluster name: [email protected]

Disk Nodes

[email protected]
[email protected]
[email protected]

Running Nodes

[email protected]
[email protected]
[email protected]

Versions

[email protected]: RabbitMQ 3.8.0 on Erlang 22.1.1
[email protected]: RabbitMQ 3.8.0 on Erlang 22.1.1
[email protected]: RabbitMQ 3.8.0 on Erlang 22.1.1

Alarms

(none)

Network Partitions

(none)

Listeners

Node: [email protected], interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: [email protected], interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: [email protected], interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: [email protected], interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: [email protected], interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: [email protected], interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: [email protected], interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: [email protected], interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: quorum_queue, state: enabled
Flag: virtual_host_metadata, state: enabled

Alternatively, list running nodes.

$ sudo rabbitmqctl cluster_status --formatter=json | jq -r .running_nodes[]
[email protected]
[email protected]
[email protected]

Example

This is an example to illustrate how RabbitMQ cluster behaves in relation to declared queues in case of node downtime.

Get rabbitmqadmin utility.

$ curl -o rabbitmqadmin http://127.0.0.1:15672/cli/rabbitmqadmin 

Ensure that executable bit is set.

$ chmod +x rabbitmqadmin 

Declare cluster_example vhost.

$ ./rabbitmqadmin --username admin --password password declare vhost name=cluster_example

Declare cluster_example user with password password, grant permissions to cluster_example vhost and set management.

$ sudo rabbitmqctl add_user cluster_example password
$ sudo rabbitmqctl set_permissions cluster_example --vhost cluster_example ".*" ".*" ".*"
$ sudo rabbitmqctl set_user_tags cluster_example  management

Declare sample exchange.

$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example declare exchange name=messages type=fanout

Declare durable queue.

$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example declare queue name=durable_messages durable=true

Declare not durable queue.

$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example declare queue name=not_durable_messages durable=false

Declare bindings.

$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example declare binding source=messages destination=durable_messages
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example declare binding source=messages destination=not_durable_messages

Install jq utility.

$ sudo apt install jq

Export configuration.

$ ./rabbitmqadmin --username admin --password password export config.json
$  cat config.json | jq .
{
  "rabbit_version": "3.8.0",
  "users": [
    {
      "name": "cluster_example",
      "password_hash": "cQb0oesCsqP6KJOjMaM36xWE9rcWef5r33peMF/5tYGGTPlR",
      "hashing_algorithm": "rabbit_password_hashing_sha256",
      "tags": "management"
    },
    {
      "name": "admin",
      "password_hash": "ZtNWgL2TdCqLMJx3jOpJVsHDsw/yiIPHF/srfFcqNkIpYzdr",
      "hashing_algorithm": "rabbit_password_hashing_sha256",
      "tags": "administrator"
    }
  ],
  "vhosts": [
    {
      "name": "cluster_example"
    },
    {
      "name": "/"
    }
  ],
  "permissions": [
    {
      "user": "cluster_example",
      "vhost": "cluster_example",
      "configure": ".*",
      "write": ".*",
      "read": ".*"
    },
    {
      "user": "admin",
      "vhost": "/",
      "configure": ".*",
      "write": ".*",
      "read": ".*"
    },
    {
      "user": "admin",
      "vhost": "cluster_example",
      "configure": ".*",
      "write": ".*",
      "read": ".*"
    }
  ],
  "topic_permissions": [],
  "parameters": [],
  "global_parameters": [
    {
      "name": "cluster_name",
      "value": "[email protected]"
    }
  ],
  "policies": [],
  "queues": [
    {
      "name": "durable_messages",
      "vhost": "cluster_example",
      "durable": true,
      "auto_delete": false,
      "arguments": {}
    },
    {
      "name": "not_durable_messages",
      "vhost": "cluster_example",
      "durable": false,
      "auto_delete": false,
      "arguments": {}
    }
  ],
  "exchanges": [
    {
      "name": "messages",
      "vhost": "cluster_example",
      "type": "fanout",
      "durable": true,
      "auto_delete": false,
      "internal": false,
      "arguments": {}
    }
  ],
  "bindings": [
    {
      "source": "messages",
      "vhost": "cluster_example",
      "destination": "durable_messages",
      "destination_type": "queue",
      "routing_key": "",
      "arguments": {}
    },
    {
      "source": "messages",
      "vhost": "cluster_example",
      "destination": "not_durable_messages",
      "destination_type": "queue",
      "routing_key": "",
      "arguments": {}
    }
  ]
}

Publish sample messages.

$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example publish exchange=messages routing_key= payload="sample message a"
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example publish exchange=messages routing_key= payload="sample message b"

Inspect queues, notice that each queue is created on a single node.

$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example list queues vhost name node messages
+-----------------+----------------------+---------------+----------+
|      vhost      |         name         |     node      | messages |
+-----------------+----------------------+---------------+----------+
| cluster_example | durable_messages     | [email protected] | 2        |
| cluster_example | not_durable_messages | [email protected] | 2        |
+-----------------+----------------------+---------------+----------+

Get sample message from queue, but do not requeue it.

$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example get  queue=durable_messages ackmode=ack_requeue_false
+-------------+----------+---------------+------------------+---------------+------------------+------------+-------------+
| routing_key | exchange | message_count |     payload      | payload_bytes | payload_encoding | properties | redelivered |
+-------------+----------+---------------+------------------+---------------+------------------+------------+-------------+
|             | messages | 1             | sample message a | 16            | string           |            | False       |
+-------------+----------+---------------+------------------+---------------+------------------+------------+-------------+

Get sample message from queue and requeue it.

$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example get  queue=durable_messages
+-------------+----------+---------------+------------------+---------------+------------------+------------+-------------+
| routing_key | exchange | message_count |     payload      | payload_bytes | payload_encoding | properties | redelivered |
+-------------+----------+---------------+------------------+---------------+------------------+------------+-------------+
|             | messages | 1             | sample message b | 16            | string           |            | True        |
+-------------+----------+---------------+------------------+---------------+------------------+------------+-------------+

Inspect queues.

$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example list queues vhost name node messages
+-----------------+----------------------+---------------+----------+
|      vhost      |         name         |     node      | messages |
+-----------------+----------------------+---------------+----------+
| cluster_example | durable_messages     | [email protected] | 1        |
| cluster_example | not_durable_messages | [email protected] | 2        |
+-----------------+----------------------+---------------+----------+

Stop the RabbitMQ node on which the above-mentioned queues reside.

$ sudo systemctl stop rabbitmq-server

Inspect queues.

$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example list queues vhost name node messages
+-----------------+------------------+---------------+----------+
|      vhost      |       name       |     node      | messages |
+-----------------+------------------+---------------+----------+
| cluster_example | durable_messages | [email protected] |          |
+-----------------+------------------+---------------+----------+

$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example get  queue=durable_messages
*** Not found: /api/queues/cluster_example/durable_messages/get

Start the RabbitMQ node on which the above-mentioned queues reside.

$ sudo systemctl start rabbitmq-server

Inspect queues.

$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example list queues vhost name node messages
+-----------------+------------------+---------------+----------+
|      vhost      |       name       |     node      | messages |
+-----------------+------------------+---------------+----------+
| cluster_example | durable_messages | [email protected] | 0        |
+-----------------+------------------+---------------+----------+

To sum up, use queue mirroring to ensure that queue contents are also protected against single node failures.

Additional notes

Clustering Guide