How to mirror RabbitMQ queues

Mirror RabbitMQ queues cluster to replicate configuration across multiple nodes.

Preliminary information

I will use RabbitMQ cluster consisting of node: rabbit, reindeer and raccoon.

Mirror RabbitMQ queues

Get rabbitmqadmin utility.

$ curl -o rabbitmqadmin http://127.0.0.1:15672/cli/rabbitmqadmin 

Ensure that executable bit is set.

$ chmod +x rabbitmqadmin 

Declare mirror_example vhost.

$ ./rabbitmqadmin --username admin --password password declare vhost name=mirror_example

Declare mirror_example user with password password, grant permissions to mirror_example vhost and set administrator.

$ sudo rabbitmqctl add_user mirror_example password
$ sudo rabbitmqctl set_permissions mirror_example --vhost mirror_example ".*" ".*" ".*"
$ sudo rabbitmqctl set_user_tags mirror_example administrator

Define mirror-all policy that will mirror every queue with name that begins with mirror-all to every node.

$ sudo rabbitmqctl set_policy -p mirror_example mirror-all "^mirror-all.*" '{"ha-mode": "all", "ha-sync-mode":"automatic"}'

Define mirror-exactly policy that will mirror every queue with name that begins with mirror-exactly to one additional node.

$ sudo rabbitmqctl set_policy -p mirror_example mirror-exactly "^mirror-exactly.*" '{"ha-mode": "exactly", "ha-params":2, "ha-sync-mode":"automatic"}'

Define mirror-nodes policy that will keep every queue with name that begins with mirror-nodes on two specified nodes.

$ sudo rabbitmqctl set_policy -p mirror_example mirror-nodes "^mirror-nodes.*" '{"ha-mode": "nodes", "ha-params": ["[email protected]", "[email protected]"], "ha-sync-mode":"automatic"}'

Display declared policies.

$ sudo rabbitmqctl list_policies -p mirror_example
Listing policies for vhost "mirror_example" ...
vhost           name            pattern                 apply-to  definition                                                                                         priority
mirror_example  mirror-nodes    ^mirror-nodes.*         all       {"ha-mode":"nodes","ha-params":["[email protected]","[email protected]"],"ha-sync-mode":"automatic"}    0
mirror_example  mirror-exactly  ^mirror-exactly.*       all       {"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"}                                     0
mirror_example  mirror-all      ^mirror-all.*           all       {"ha-mode":"all","ha-sync-mode":"automatic"}                                                       0

Declare sample exchange.

$ ./rabbitmqadmin --username mirror_example --password password --vhost mirror_example declare exchange name=messages type=fanout

Declare mirror-all.messages queue.

$ ./rabbitmqadmin --username mirror_example --password password --vhost mirror_example declare queue name=mirror-all.messages durable=true

Declare mirror-exactly.messages queue.

$ ./rabbitmqadmin --username mirror_example --password password --vhost mirror_example declare queue name=mirror-exactly.messages durable=true

Declare mirror-nodes.messages queue.

$ ./rabbitmqadmin --username mirror_example --password password --vhost mirror_example declare queue name=mirror-nodes.messages durable=true

Declare bindings.

$ ./rabbitmqadmin --username mirror_example --password password --vhost mirror_example declare binding source=messages destination=mirror-all.messages
$ ./rabbitmqadmin --username mirror_example --password password --vhost cluster_example declare binding source=messages destination=mirror-exactly.messages
$ ./rabbitmqadmin --username mirror_example --password password --vhost cluster_example declare binding source=messages destination=mirror-nodes.messages

Publish sample message.

$ ./rabbitmqadmin --username mirror_example --password password --vhost mirror_example publish exchange=messages routing_key= payload="sample message a"

Inspect queues, notice that each queue is created on a single node.

$ ./rabbitmqadmin --username mirror_example  --password password --vhost mirror_example list queues vhost name node slave_nodes effective_policy_definition.ha-mode effective_policy_definition.ha-params  messages durable
+----------------+--------------------------+----------------+---------------------------------------+-------------------------------------+---------------------------------------+----------+---------+
|     vhost      |           name           |      node      |              slave_nodes              | effective_policy_definition.ha-mode | effective_policy_definition.ha-params | messages | durable |
+----------------+--------------------------+----------------+---------------------------------------+-------------------------------------+---------------------------------------+----------+---------+
| mirror_example | mirror-all.messages      | [email protected]  | ["[email protected]", "[email protected]"] | all                                 |                                       | 1        | True    |
| mirror_example | mirror-exactly.messages  | [email protected]  | ["[email protected]"]                   | exactly                             | 2                                     | 1        | True    |
| mirror_example | mirror-nodes.messages    | [email protected] | ["[email protected]"]                   | nodes                               | ["[email protected]", "[email protected]"] | 1        | True    |
+----------------+--------------------------+----------------+---------------------------------------+-------------------------------------+---------------------------------------+----------+---------+

Stop the RabbitMQ node on which the above-mentioned queues reside.

$ sudo systemctl stop rabbitmq-server

Inspect queues.

$ ./rabbitmqadmin --username mirror_example  --password password --vhost mirror_example list queues vhost name node slave_nodes effective_policy_definition.ha-mode effective_policy_definition.ha-params  messages durable
+----------------+--------------------------+-----------------+---------------------+-------------------------------------+---------------------------------------+----------+---------+
|     vhost      |           name           |      node       |     slave_nodes     | effective_policy_definition.ha-mode | effective_policy_definition.ha-params | messages | durable |
+----------------+--------------------------+-----------------+---------------------+-------------------------------------+---------------------------------------+----------+---------+
| mirror_example | mirror-all.messages      | [email protected]  | ["[email protected]"] | all                                 |                                       | 1        | True    |
| mirror_example | mirror-exactly.messages  | [email protected] | ["[email protected]"]  | exactly                             | 2                                     | 1        | True    |
| mirror_example | mirror-nodes.messages    | [email protected]  | ["[email protected]"] | nodes                               | ["[email protected]", "[email protected]"] | 1        | True    |
+----------------+--------------------------+-----------------+---------------------+-------------------------------------+---------------------------------------+----------+---------+

Start the stopped RabbitMQ node.

$ sudo systemctl start rabbitmq-server

Inspect queues.

$ ./rabbitmqadmin --username mirror_example  --password password --vhost mirror_example list queues vhost name node slave_nodes effective_policy_definition.ha-mode effective_policy_definition.ha-params  messages durable
+----------------+--------------------------+-----------------+--------------------------------------+-------------------------------------+---------------------------------------+----------+---------+
|     vhost      |           name           |      node       |             slave_nodes              | effective_policy_definition.ha-mode | effective_policy_definition.ha-params | messages | durable |
+----------------+--------------------------+-----------------+--------------------------------------+-------------------------------------+---------------------------------------+----------+---------+
| mirror_example | mirror-all.messages      | [email protected]  | ["[email protected]", "[email protected]"] | all                                 |                                       | 1        | True    |
| mirror_example | mirror-exactly.messages  | [email protected] | ["[email protected]"]                   | exactly                             | 2                                     | 1        | True    |
| mirror_example | mirror-nodes.messages    | [email protected]  | ["[email protected]"]                  | nodes                               | ["[email protected]", "[email protected]"] | 1        | True    |
+----------------+--------------------------+-----------------+--------------------------------------+-------------------------------------+---------------------------------------+----------+---------+

Install jq utility.

$ sudo apt install jq

Export configuration.

$ ./rabbitmqadmin --username admin --password password export config.json
$ cat config.json | jq .
{
  "rabbit_version": "3.8.0",
  "users": [
    {
      "name": "admin",
      "password_hash": "m4Gwgzra8yauPPLYAIKlAAE08uCs2sPWQsY/ncND0GKwLVgT",
      "hashing_algorithm": "rabbit_password_hashing_sha256",
      "tags": "administrator"
    },
    {
      "name": "mirror_example",
      "password_hash": "/z+hiB2k4CtaYBKYvt2nqQwoBImCU75pMIw4cBkX6Tsr5ZDp",
      "hashing_algorithm": "rabbit_password_hashing_sha256",
      "tags": "administrator"
    }
  ],
  "vhosts": [
    {
      "name": "/"
    },
    {
      "name": "mirror_example"
    }
  ],
  "permissions": [
    {
      "user": "admin",
      "vhost": "/",
      "configure": ".*",
      "write": ".*",
      "read": ".*"
    },
    {
      "user": "mirror_example",
      "vhost": "mirror_example",
      "configure": ".*",
      "write": ".*",
      "read": ".*"
    },
    {
      "user": "admin",
      "vhost": "mirror_example",
      "configure": ".*",
      "write": ".*",
      "read": ".*"
    }
  ],
  "topic_permissions": [],
  "parameters": [],
  "global_parameters": [
    {
      "name": "cluster_name",
      "value": "[email protected]"
    }
  ],
  "policies": [
    {
      "vhost": "mirror_example",
      "name": "mirror-all",
      "pattern": "^mirror-all.*",
      "apply-to": "all",
      "definition": {
        "ha-mode": "all",
        "ha-sync-mode": "automatic"
      },
      "priority": 0
    },
    {
      "vhost": "mirror_example",
      "name": "mirror-exactly",
      "pattern": "^mirror-exactly.*",
      "apply-to": "all",
      "definition": {
        "ha-mode": "exactly",
        "ha-params": 2,
        "ha-sync-mode": "automatic"
      },
      "priority": 0
    },
    {
      "vhost": "mirror_example",
      "name": "mirror-nodes",
      "pattern": "^mirror-nodes.*",
      "apply-to": "all",
      "definition": {
        "ha-mode": "nodes",
        "ha-params": [
          "[email protected]",
          "[email protected]"
        ],
        "ha-sync-mode": "automatic"
      },
      "priority": 0
    }
  ],
  "queues": [
    {
      "name": "mirror-exactly.messages",
      "vhost": "mirror_example",
      "durable": true,
      "auto_delete": false,
      "arguments": {}
    },
    {
      "name": "mirror-all.messages",
      "vhost": "mirror_example",
      "durable": true,
      "auto_delete": false,
      "arguments": {}
    },
    {
      "name": "mirror-nodes.messages",
      "vhost": "mirror_example",
      "durable": true,
      "auto_delete": false,
      "arguments": {}
    }
  ],
  "exchanges": [
    {
      "name": "messages",
      "vhost": "mirror_example",
      "type": "fanout",
      "durable": true,
      "auto_delete": false,
      "internal": false,
      "arguments": {}
    }
  ],
  "bindings": [
    {
      "source": "messages",
      "vhost": "mirror_example",
      "destination": "mirror-all.messages",
      "destination_type": "queue",
      "routing_key": "",
      "arguments": {}
    },
    {
      "source": "messages",
      "vhost": "mirror_example",
      "destination": "mirror-exactly.messages",
      "destination_type": "queue",
      "routing_key": "",
      "arguments": {}
    },
    {
      "source": "messages",
      "vhost": "mirror_example",
      "destination": "mirror-nodes.messages",
      "destination_type": "queue",
      "routing_key": "",
      "arguments": {}
    }
  ]
}

Additional notes

Clustering Guide

Highly Available (Mirrored) Queues