Memory Alarms
Overview
This guide covers RabbitMQ memory threshold and paging settings, running nodes on 64-bit and 32-bit systems, and other related topics.
A separate guide, Reasoning About Memory Use, covers how to determine what consumes memory on a running RabbitMQ node for the purpose of monitoring or troubleshooting.
Portions of this guide related to queue content paging to disk are obsolete or not applicable for quorum queues, streams and classic queues storage version 2 (CQv2). All of them actively move data to disk and do not generally accumulate a significant backlog of messages in memory.
Memory Threshold: What it is and How it Works
The RabbitMQ server detects the total amount of RAM installed in the computer on startup and when
rabbitmqctl set_vm_memory_high_watermark fraction is
executed. By default, when the RabbitMQ server uses above 40%
of the available RAM, it raises a memory alarm and blocks all
connections that are publishing messages. Once the memory alarm has cleared (e.g. due
to the server paging messages to disk or delivering them to
clients that consume and acknowledge the deliveries) normal service resumes.
The default memory threshold is set to 40% of installed RAM. Note that this does not prevent the RabbitMQ server from using more than 40%, it is merely the point at which publishers are throttled. Erlang's garbage collector can, in the worst case, cause double the amount of memory to be used (by default, 80% of RAM). It is strongly recommended that OS swap or page files are enabled.
32-bit architectures tend to impose a per process memory limit of 2GB. Common implementations of 64-bit architectures (i.e. AMD64 and Intel EM64T) permit only a paltry 256TB per process. 64-bit Windows further limits this to 8TB. However, note that even under 64-bit OSes, a 32-bit process frequently only has a maximum address space of 2GB.
Configuring the Memory Threshold
The memory threshold at which the flow control is triggered can be adjusted by editing the configuration file.
The example below sets the threshold to the default value of 0.4:
\# new style config format, recommended
vm_memory_high_watermark.relative = 0.4
The default value of 0.4 stands for 40% of available (detected) RAM or 40% of available virtual address space, whichever is smaller. E.g. on a 32-bit platform with 4 GiB of RAM installed, 40% of 4 GiB is 1.6 GiB, but 32-bit Windows normally limits processes to 2 GiB, so the threshold is actually to 40% of 2 GiB (which is 820 MiB).
Alternatively, the memory threshold can be adjusted by setting an absolute limit of RAM used by the node. The example below sets the threshold to 1073741824 bytes (1024 MiB):
vm_memory_high_watermark.absolute = 1073741824
Same example, but using memory units:
vm_memory_high_watermark.absolute = 1024MiB
If the absolute limit is larger than the installed RAM or available virtual address space, the threshold is set to whichever limit is smaller.
The memory limit is appended to the log file when the RabbitMQ node starts:
2019-06-10 23:17:05.976 [info] <0.308.0> Memory high watermark set to 1024 MiB (1073741824 bytes) of 8192 MiB (8589934592 bytes) total
The memory limit may also be queried using the
rabbitmq-diagnostics memory_breakdown and rabbitmq-diagnostics status commands.
The threshold can be changed while the broker is running using the
rabbitmqctl set_vm_memory_high_watermark <em><fraction></em>
command or
rabbitmqctl set_vm_memory_high_watermark absolute <em><memory_limit></em>
For example:
rabbitmqctl set_vm_memory_high_watermark 0.6
and
rabbitmqctl set_vm_memory_high_watermark absolute "4G"
When using the absolute mode, it is possible to use one of the following memory units:
M,MiBfor mebibytes (2^20bytes)MBfor megabytes (10^6bytes)G,GiBfor gibibytes (2^30bytes)GBfor gigabytes (10^9bytes)
Both commands will have an effect until the node stops. To make the setting survive node restart, use the configuration setting instead.
The memory limit may change on systems with hot-swappable RAM when this command is executed without altering the threshold, due to the fact that the total amount of system RAM is queried.
Stop All Publishing
When the threshold or absolute limit is set to 0, it makes the memory alarm go off
immediately and thus eventually blocks all publishing connections. This may be
useful if you wish to deactivate publishing globally:
rabbitmqctl set_vm_memory_high_watermark 0
Limited Address Space
When running RabbitMQ inside a 32 bit Erlang VM in a 64 bit OS (or a 32 bit OS with PAE), the addressable memory is limited. The server will detect this and log a message like:
2018-11-22 10:44:33.654 [warning] Only 2048MB of 12037MB memory usable due to limited address space.
The memory alarm system is not perfect. While stopping publishing will usually prevent any further memory from being used, it is quite possible for other things to continue to increase memory use. Normally when this happens and the physical memory is exhausted the OS will start to swap. But when running with a limited address space, running over the limit will cause the VM to terminate or killed by an out-of-memory mechanism of the operating system.
It is therefore strongly recommended to run RabbitMQ on a 64 bit OS and a 64-bit Erlang runtime.
Configuring the Paging Threshold
This section is obsolete or not applicable for quorum queues, streams and classic queues storage version 2 (CQv2). All of them actively move data to disk and do not generally accumulate a significant backlog of messages in memory.
Before the broker hits the high watermark and blocks publishers, it will attempt to free up memory by instructing CQv1 queues to page their contents out to disc. Both persistent and transient messages will be paged out (the persistent messages will already be on disc but will be evicted from memory).
By default this starts to happen when the broker is 50% of
the way to the high watermark (i.e. with a default high
watermark of 0.4, this is when 20% of memory is used). To
change this value, modify
the vm_memory_high_watermark_paging_ratio
configuration from its default value
of 0.5. For example:
vm_memory_high_watermark_paging_ratio = 0.75
vm_memory_high_watermark.relative = 0.4
The above configuration starts paging at 30% of memory used, and blocks publishers at 40%.
Unrecognised Platforms
If the RabbitMQ server is unable to detect the operating system it is running on, it will append a warning to the log file. It then assumes than 1GB of RAM is installed:
2018-11-22 10:44:33.654 [warning] Unknown total memory size for your OS {unix,magic_homegrown_os}. Assuming memory size is 1024MB.
In this case, the vm_memory_high_watermark
configuration value is used to scale the assumed 1GB
RAM. With the default value of
vm_memory_high_watermark set to 0.4,
RabbitMQ's memory threshold is set to 410MB, thus it will
throttle producers whenever RabbitMQ is using more than
410MB memory. Thus when RabbitMQ can't recognize your
platform, if you actually have 8GB RAM installed and you
want RabbitMQ to throttle producers when the server is using
above 3GB, set vm_memory_high_watermark to 3.
For guidelines on recommended RAM watermark settings, see Production Checklist.