Securing Cluster (Inter-node) and CLI Tool Communication with TLS

Overview

RabbitMQ nodes accept connections from clients as well as peer cluster nodes and CLI tools.

The main TLS and Troubleshooting TLS guides explain how to secure client connections with TLS. It may be desired to add a layer of encryption and an extra layer of authentication to the other two kinds of connections. This guide explains how to do that.

Switching inter-node and CLI tool communication requires configuring a few runtime flags. They provide the node with a CA certificate bundle and a certificate/key pair. CLI tools also have to be configured to use a certificate/key pair as TLS-enabled nodes won't accept unencrypted connections from CLI tools and peers.

This guide assumes the reader is familiar with the basics of TLS and peer verification (authentication) covered in the main TLS guide.

It also assumes that you already have a CA certificate bundle and a certificate/key pair generated for every cluster node and every host CLI tools will be use on. In production environments those certificates will often be produced by operators or deployment tools. For development and experimentation, there is a quick way to generate them using OpenSSL and Python.

This guide will reference three files:

ca_certificate.pem: a certificate authority bundle
server_certificate.pem: a certificate (public key) that will be used by the configured node (and/or CLI tools)
server_key.pem: a private key that will be used by the configured node (and/or CLI tools)

Make sure you have them ready before we start.

The Basics

Configuring a node to communicate over TLS-enabled connections involves a few steps. With supported Erlang versions there are two ways of doing it.

The steps are very similar on all operating systems supported but minor details will be different on Windows due to a different shell language.

Strategy one involves the following steps:

Tell the node to use encrypted inter-node connections using a runtime flag, -proto_dist inet_tls
Combine public and private keys to be used by the node into a single file
Tell the node where to find its certificate and private key using another runtime flag, -ssl_dist_opt server_certfile
Tell the node about any additional TLS settings desired, using other -ssl_dist_opt options, for example: -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true to enable secure renegotiation

Strategy two is very similar but instead of specifying a set of runtime flags, those options can be specified in a file similar to RabbitMQ's advanced.config file and the runtime will be pointed at that file. Therefore the steps are:

Tell the node to use encrypted inter-node connections using a runtime flag, -proto_dist inet_tls
Deploy an inter-node TLS settings file that contains information about certificate/key pair locations, CA bundle location, TLS settings used and so on
Tell the node where to find its inter-node TLS setting file using another runtime flag, -ssl_dist_optfile

We encourage operators to choose the strategy that works best for their deployment tools of choice.

With both options environment variables are used to pass those options to the runtime. This is best done using rabbitmq-env.conf as explained in the Configuration guide.

Once a node has inter-node connection configured with TLS, CLI tools such as rabbitmqctl and rabbitmq-diagnostics also must use TLS to talk to the node. Plain TCP connections will be fail.

Deploying Inter-node TLS

Once the certificate/key pair files and configuration are in place the new node can be started. Note that it might be necessary to first stop the node, then deploy the files and configuration, and finally start the node. This is because CLI tools configured to use TLS won't be able to connect to a node that does not expect TLS-enabled CLI tool connections.

For nodes and CLI tools to perform TLS handshake and peer verification successfully, the same peer verification example, certificate/key pairs used by other nodes and CLI tools must be signed by the same certificate authority as the initial node or a different CA that is trusted on all cluster nodes.

This is no different from how peer verification works for client and plugin TLS connections.

It is possible to reuse a single certificate/key pair for all nodes and CLI tools. The certificate can also use a wildcard Subject Alternative Name (SAN) or Common Name (CN) such as *.rabbitmq.example.local that would match every hostname in the cluster.

Strategy One (Using Individual Flags) on Linux, macOS and BSD

Combining Certificate and Private Key

The first strategy covered in this guide requires node's public and private keys to be combined into a single file. Let's call it a combined keys file. To combined them, simply concatenate the private key file, server_key.pem in the example below, to the end of the public key file, server_certificate.pem, starting with a new line:

cat server_certificate.pem server_key.pem &gt; combined_keys.pem

This can be done using a text editor and not just command line tools such as cat.

Configuring Individual Runtime Flags for Inter-node TLS

Assuming a combined keys file from the section above is ready, next we infer the Erlang TLS library path and export ERL_SSL_PATH in rabbitmq-env.conf to point at it:

# These commands ensure that `ERL_SSL_PATH` is the first line in
# /etc/rabbitmq/rabbitmq-env.conf and will preserve the existing
# contents of that file if it already exists

erl -noinput -eval 'io:format("ERL_SSL_PATH=~s~n", [filename:dirname(code:which(inet_tls_dist))])' -s init stop &gt; /tmp/ssl-path.txt
cat /tmp/ssl-path.txt /etc/rabbitmq/rabbitmq-env.conf &gt; /tmp/new-rabbitmq-env.conf
mv -f /tmp/new-rabbitmq-env.conf /etc/rabbitmq/rabbitmq-env.conf

This makes it possible for the node to load a module, inet_tls_dist, which is used for encrypted inter-node communication, from the path.

Step number two is telling the runtime to use that module using the -proto_dist inet_tls runtime flag. As with other runtime flags, SERVER_ADDITIONAL_ERL_ARGS is the most convenient and compatible to pass them.

Please note that the double quotes must be used here because the environment variable value is multi-line:

# -pa $ERL_SSL_PATH prepends the directory ERL_SSL_PATH points at to the code path
# -proto_dist inet_tls tells the runtime to encrypt inter-node communication
# -ssl_dist_opt server_certfile /path/to/combined_keys.pem tells the runtime
#                               where to find the combined certificate/key file
# -ssl_dist_opt server_password password   required if the private key is encrypted
#
SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH \
  -proto_dist inet_tls \
  -ssl_dist_opt server_certfile /path/to/combined_keys.pem \
  -ssl_dist_opt server_password password

Next step is to build on the previous example and enable secure renegotiation for inter-node TLS connections. While this is optional, it is highly recommended. The same -ssl_dist_opt can be used to enable more TLS-related settings. They won't be covered in this example:

# -pa $ERL_SSL_PATH prepends the directory ERL_SSL_PATH points at to the code path
# -proto_dist inet_tls tells the runtime to encrypt inter-node communication
# -ssl_dist_opt server_certfile /path/to/combined_keys.pem tells the runtime
#                               where to find the combined certificate/key file
# -ssl_dist_opt server_password password   required if the private key is encrypted
# -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true enables an additional TLS setting: secure renegotiation
SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH \
  -proto_dist inet_tls \
  -ssl_dist_opt server_certfile /path/to/combined_keys.pem \
  -ssl_dist_opt server_password password \
  -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"

Once a node has inter-node connection configured with TLS, CLI tools such as rabbitmqctl and rabbitmq-diagnostics also must use TLS to talk to the node. Plain TCP connections will be fail.

This is done very similarly to what the example above does using SERVER_ADDITIONAL_ERL_ARGS but this time the environment variable is RABBITMQ_CTL_ERL_ARGS. It controls runtime flags used by CLI tools.

Here is the complete /etc/rabbitmq/rabbitmq-env.conf file:

# IMPORTANT:
# the following path is system dependent (will
# change depending on the Erlang version, distribution,
# and installation method used). Please double check it before proceeding!
ERL_SSL_PATH="/usr/lib64/erlang/lib/ssl-9.4/ebin"

# -pa $ERL_SSL_PATH prepends the directory ERL_SSL_PATH points at to the code path
# -proto_dist inet_tls tells the runtime to encrypt inter-node communication
# -ssl_dist_opt server_certfile /path/to/combined_keys.pem tells the runtime
#                               where to find the combined certificate/key file
# -ssl_dist_opt server_password password   required if the private key is encrypted
# -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true enables an additional TLS setting: secure renegotiation
SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH \
  -proto_dist inet_tls \
  -ssl_dist_opt server_certfile /path/to/combined_keys.pem \
  -ssl_dist_opt server_password password \
  -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"

# Same settings as above but for CLI tools
RABBITMQ_CTL_ERL_ARGS="-pa $ERL_SSL_PATH \
  -proto_dist inet_tls \
  -ssl_dist_opt server_certfile /path/to/combined_keys.pem \
  -ssl_dist_opt server_password password \
  -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"

Strategy Two (Using a Single TLS Option File) on Linux, macOS and BSD

Using a Separate Setting File for Inter-node TLS

Modern Erlang versions support a runtime flag, -ssl_dist_optfile, that can be used to configure TLS for inter-node communication using a single file. This simplifies the arguments passed on the command line itself.

Here is a complete /etc/rabbitmq/rabbitmq-env.conf file using this setting. Note that the name of the -ssl_dist_optfile file is not significant but it must be stored in a location readable by the effective rabbitmq user:

# NOTE: the following path is system dependent and will change between Erlang
#       versions
ERL_SSL_PATH="/usr/lib64/erlang/lib/ssl-9.4/ebin"

# -pa $ERL_SSL_PATH prepends the directory ERL_SSL_PATH points at to the code path
# -proto_dist inet_tls tells the runtime to encrypt inter-node communication
# -ssl_dist_optfile tells the runtime where to find its inter-node TLS configuration file
SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH
  -proto_dist inet_tls
  -ssl_dist_optfile /etc/rabbitmq/inter_node_tls.config"

RABBITMQ_CTL_ERL_ARGS="-pa $ERL_SSL_PATH
  -proto_dist inet_tls
  -ssl_dist_optfile /etc/rabbitmq/inter_node_tls.config"

Here is an example /etc/rabbitmq/inter_node_tls.config file that uses separate server certificate and private key files, enables peer verification and requires peers to present a certificate:

[
  {server, [
    {cacertfile, "/full/path/to/ca_certificate.pem"},
    {certfile,   "/full/path/to/server_certificate.pem"},
    {keyfile,    "/full/path/to/server_key.pem"},
    {password,   "password-if-keyfile-is-encrypted"},
    {secure_renegotiate, true},
    {verify, verify_peer},
    {fail_if_no_peer_cert, true}
  ]},
  {client, [
    {cacertfile, "/full/path/to/ca_certificate.pem"},
    {certfile,   "/full/path/to/client_certificate.pem"},
    {keyfile,    "/full/path/to/client_key.pem"},
    {password,   "password-if-keyfile-is-encrypted"},
    {secure_renegotiate, true},
    {verify, verify_peer}
  ]}
].

These options are documented further in the Erlang/OTP documentation.

Windows

Both strategies covered above for Linux, macOS and BSD systems can be used on Windows. All fundamentals are the same.

There are, however, some minor differences specific to Windows. First, the command that outputs the location of the inet_tls_dist module is different due to Windows shell parsing rules. it looks like this

erl -noinput -eval "io:format(""ERL_SSL_PATH=~s~n"", [filename:dirname(code:which(inet_tls_dist))])" -s init stop

Next, the file containing the custom environment variables is named rabbitmq-env-conf.bat on Windows. This file must be saved to the %AppData%\RabbitMQ directory of the administrative user that installed RabbitMQ.

Here is a complete rabbitmq-env-conf.bat file using the -ssl_dist_opfile setting (strategy two covered above). Note the use of forward-slash directory delimiters.

@echo off
rem NOTE: If spaces are present in any of these paths,
rem double quotes must be used.

rem NOTE: the following path is **system dependent** and will vary between Erlang versions
rem       and installation paths
set SSL_PATH="C:/Program Files/erl10.0.1/lib/ssl-9.0/ebin"

rem -pa $ERL_SSL_PATH prepends the directory ERL_SSL_PATH points at to the code path
rem -proto_dist inet_tls tells the runtime to encrypt inter-node communication
rem -ssl_dist_optfile tells the runtime where to find its inter-node TLS configuration file
set SERVER_ADDITIONAL_ERL_ARGS=-pa %SSL_PATH% ^
    -proto_dist inet_tls ^
    -ssl_dist_optfile C:/Users/rmq_user/AppData/Roaming/RabbitMQ/inter_node_tls.config

rem Same as above but for CLI tools
set CTL_ERL_ARGS=-pa %SSL_PATH% ^
    -proto_dist inet_tls ^
    -ssl_dist_optfile C:/Users/rmq_user/AppData/Roaming/RabbitMQ/inter_node_tls.config

Below is an example inter_node_tls.config file. As with other operating systems, more TLS options are available to be set if necessary.

[
    {server, [
        {cacertfile, "C:/Path/To/ca_certificate.pem"},
        {certfile, "C:/Path/To/server_certificate.pem"},
        {keyfile, "C:/Path/To/server_key.pem"},
        {password, "password-if-keyfile-is-encrypted"},
        {secure_renegotiate, true},
        {verify, verify_peer},
        {fail_if_no_peer_cert, true}
    ]},
    {client, [
        {cacertfile, "C:/Path/To/ca_certificate.pem"},
        {certfile, "C:/Path/To/client_certificate.pem"},
        {keyfile, "C:/Path/To/client_key.pem"},
        {password, "password-if-keyfile-is-encrypted"},
        {secure_renegotiate, true},
        {verify, verify_peer}
    ]}
].

Securing Cluster (Inter-node) and CLI Tool Communication with TLS

Overview​

The Basics​

Deploying Inter-node TLS​

Strategy One (Using Individual Flags) on Linux, macOS and BSD​

Combining Certificate and Private Key​

Configuring Individual Runtime Flags for Inter-node TLS​

Strategy Two (Using a Single TLS Option File) on Linux, macOS and BSD​

Using a Separate Setting File for Inter-node TLS​

Windows​