Loading ...
Eric Asberry
Other
Dev Stuff
22
0
Try Now
Log In
Pricing
Understanding memory usage in Docker Desktop on Mac There have been many reports of possible memory leaks and apparent high memory usage of Docker Desktop since the upgrade to macOS 10.14 (Mojave). This document summarises the investigation into these reports and includes recommendations to help users better understand the memory usage of the system. Conclusions 1. The memory used by Docker Desktop does not grow over time, i.e., there is no memory leak. 2. The system is able to recover memory from Docker Desktop and give it to other processes if the system is under memory pressure. This is not reflected in the headline memory figure in the Activity Monitor, but can be seen by looking at the Real Memory. 3. The headline memory figure also has a double-counting bug in MacOS Mojave, causing it to report double the actual memory allocated. We have reported this bug to Apple. Recommendations 1. Add and then monitor the “Real Mem” column in Activity Monitor which counts exactly how much physical memory a process is taking right now. This value will move up and down depending on which part of the system needs the memory most. 2. Ignore the “Memory” column (the default column) in Activity Monitor which counts VM memory twice due to a bug in macOS 10.14 (Mojave) which has been reported to Apple. Understanding memory on the Mac Modern operating systems like macOS use “virtual” memory. Each process has its own separate “address space” which can be much larger than the amount of physical memory in the machine. When a process allocates memory (e.g. via calling malloc(3)), macOS will simply mark a range of virtual addresses as used but it will not actually allocate any physical memory. Physical memory is allocated on-demand, when the virtual memory is first read or written. When some physical memory hasn’t been used for a while and the machine is under “memory pressure” (i.e. it is nearly full), macOS will save the contents of the memory to disk (a process known as “swapping”) so the physical memory can be recycled. If the process needs the memory later, macOS will re-load it. More details of virtual memory on macOS are available in the official documentation. It is a common pattern for an application to “allocate” a large amount of memory and then only use parts of it at a time. The OS will only load memory that is actually needed into physical memory. Therefore to understand the true impact of a process on the system we need to watch how much memory the process is actually using, not the total it has “allocated”. The following sections describe tools that we can use to understand the memory usage of processes on macOS. Activity Monitor Activity Monitor is a GUI application which displays live CPU, memory, disk and network I/O information. On the “Memory” screen either double-click on the row to view the Real Memory or add the additional column “Real Memory” from the View / Columns menu: Activity Monitor would then look like this: The “Memory” column is a total of all memory which the application has allocated and written to at least once. Note that this never decreases unless an application officially deallocates memory by calling free(3), even if the memory is actually not still being used. The “Real Mem” column (which we recommend adding) shows the current amount of physical memory the process is using at this moment in time. This will increase when the application uses more memory and decrease when macOS recycles it for some other purpose. The “Memory Pressure” graph at the bottom of the window indicates how busy the memory system is. In the screenshot above the line on the graph is near the bottom and coloured green, indicating the system is not under pressure. Next to the graph are system-wide summaries of how memory is being used. Note that the system will try to use free memory for useful things like caching, so we expect most memory to be always be used. Command-line: ps The command-line utility ps can be used to query memory information, for example: $ ps -x -o pid,rss,vsz,command PID RSS VSZ COMMAND 328 3164 4305816 /usr/sbin/cfprefsd agent 329 5896 4334364 /usr/libexec/UserEventAgent (Aqua) … This displays the process id (PID), the Resident Set Size (RSS) (equivalent to “Real Mem” in Activity Monitor), Virtual Size (VSZ) (the size of the virtual address space of the process) and the command name. Unfortunately there is no ps option which corresponds to the “Memory” column in Activity Monitor. Command-line: vmmap The command-line utility vmmap displays and summarises the virtual memory allocated by a process, for example: $ vmmap -summary com.docker.hyperkit Process: com.docker.hyperkit [74205] Path: /Applications/Docker.app/Contents/Resources/bin/com.docker.hyperkit ... Physical footprint: 7.8G Physical footprint (peak): 7.8G ... The “Physical footprint” corresponds to the “Memory” column in Activity Monitor. This is new in macOS 10.14 (Mojave). Is there a memory leak? To discover whether there is a memory leak, we must monitor the memory usage over time. A leak would manifest as a steady increase in memory usage with no upper limit. Note that memory usage being high does not by itself demonstrate that a leak is present; we must show the increase over time. Experiment We installed Docker Desktop on a MacBook Pro with macOS 10.14 (Mojave). We enabled Kubernetes and then monitored the “Memory” and “Real Mem” values using a combination of ps and vmmap using this code. Results The following graph shows the memory usage over an approximately 2 day long period: Note that the reported “Memory” in Activity Monitor remains constant at just under 4GB but the “Real Mem” (which represents how much physical memory is actually being used) decreases a little over time. Conclusion There is no sign of a memory leak while running an idle Docker and Kubernetes on Mac. What happens if there is memory pressure? To understand what happens when the machine is running low on memory (under “memory pressure”) we must monitor Docker Desktop while running other memory-intensive applications. Experiment We installed Docker Desktop on a MacBook Pro with macOS 10.14 (Mojave). We then executed the following series of steps: ● For both 2GB and 4GB VM memory settings ○ Start Docker Desktop (with the default 1GB of swap) ○ Repeat a few times: ■ Run a container which allocates and uses as much VM memory as possible (docker run -it --privileged djs55/write-all-memory) ■ Stop the container ■ Run a program which allocates and uses 15500 MiB on the host (which has 16GiB total), to cause memory pressure on the host ■ Stop the program Results The following graph shows the memory usage of hyperkit when the VM is set to use 2GB in Docker Desktop settings: Notice that the “Memory” in Activity Monitor rises to a maximum value (4GB) before remaining constant. The “Real Mem” (which represents how much physical memory is actually being used) goes up and down, depending on whether memory is needed inside the VM or on the host. The following graph shows the memory usage of hyperkit when the VM is set to use 4GB in Docker Desktop settings: Notice that the “Memory” in Activity Monitor rises to a maximum value (8GB) before remaining constant. The “Real Mem” (which represents how much physical memory is actually being used) goes up and down, depending on whether memory is needed inside the VM or on the host. Conclusion The “Real Mem” graph shows that macOS virtual memory system is able to move memory between the host and the VM, depending on where it is most needed. The graphs show that macOS can decrease the VM memory to the same level (approximately 1GB) irrespective of the VM memory setting in Docker Desktop preferences i.e. all possible VM memory is successfully freed and recycled. The misleading “Memory” column in Activity Monitor The results from the earlier experiments demonstrate the “Real Mem” behaves as we would expect: when the VM needs more memory “Real Mem” goes up to the maximum given in Docker Desktop settings; when the host needs more memory it goes down. However the “Memory” in “Activity Monitor” is behaving strangely: “Memory” seems to always increase, up to a maximum of 2x the memory size in Docker Desktop settings. To understand why this is happening we need to modify hyperkit itself. Experiment The following patch was applied to hyperkit: diff --git a/src/lib/vmm/vmm_mem.c b/src/lib/vmm/vmm_mem.c index 2c5c858..eb1c8e7 100644 --- a/src/lib/vmm/vmm_mem.c +++ b/src/lib/vmm/vmm_mem.c @@ -47,11 +48,13 @@ vmm_mem_alloc(uint64_t gpa, size_t size) void *object; object = valloc(size); if (!object) { xhyve_abort("vmm_mem_alloc failed\n"); } - + sleep(30); + /* Write to the region from the host to cause all pages to be allocated */ + memset(object, 0, size); + sleep(30); if (hv_vm_map(object, gpa, size, HV_MEMORY_READ | HV_MEMORY_WRITE | HV_MEMORY_EXEC)) { The vmm_mem_alloc function is called to allocate memory for the VM. The memory is first allocated with valloc(3) on the host and then mapped into the VM using the function hv_vm_map in the hypervisor framework. The patch calls to memset(3) to write zeroes to the allocated region to make sure macOS allocates physical memory (recall physical memory is only allocated on-demand) and the calls to sleep(3) insert 30 second delays to make the effects of the allocation more obvious in Activity Monitor. We executed the following series of steps: ● For both macOS 10.12 and 10.14 ○ Start Docker Desktop with the modified hyperkit binary ○ Run a container which allocates and uses as much VM memory as possible (docker run -it --privileged djs55/write-all-memory). Results The following graph shows the “Memory” and “Real Mem” in Activity Monitor as the modified hyperkit starts, allocates memory and the VM boots on macOS 10.12 (Note the same behaviour is also seen on 10.13): The 4 GB of VM memory is allocated in two chunks: the first is of size 3 GB and the second is of size 1 GB. When the memory is allocated with valloc(3) and written to with memset(3) both the “Memory” and “Real Mem” increase in Activity Monitor as expected. When the VM starts booting and writing to the already-allocated memory, the “Memory” and “Real Mem” stay constant at 4 GB in Activity Monitor. The following graph shows the “Memory” and “Real Mem” in Activity Monitor as the modified hyperkit starts, allocates memory and the VM boots on macOS 10.14 (Mojave): As with macOS 10.12 and 10.13, we observe that the 4 GB of memory is allocated in two chunks: the first is of size 3 GB and the second is of 1 GB. When the memory is allocated with valloc(3) and written to with memset(3) both the “Memory” and “Real Mem” increase in Activity Monitor as expected. However when the VM starts booting and starts writing to the already-allocated memory, the “Memory” in Activity monitor increases again but the “Real Mem” does not. When the container in the VM writes to all the VM memory the “Memory” in Activity Monitor is 8 GB, 2x the 4 GB setting in Docker Desktop and 2x the “Memory” shown in macOS 10.12 and 10.13. “Real Mem” remains at 4 GB. Conclusion It seems that Activity Monitor “Memory” is double-counting the VM memory in macOS 10.14. The memory is counted once when the hyperkit process allocates and writes to it and then counted a second time when the VM writes to it, even though it is the same memory. This means that the “Memory” figure in Activity Monitor is often double the actual memory allocated. The “Real Mem” in Activity Monitor is behaving as expected. Locating the bug The previous experiment shows that the “Memory” column in Activity Monitor appears to double-count memory shared with running VMs. Is the bug in hyperkit? Is hyperkit using the macOS virtualisation APIs incorrectly? Or is the bug in macOS itself? To determine whether the bug is in hyperkit or not, we can compare it to qemu: another open-source hypervisor with a completely independent codebase. Experiment We first created a small bootable test VM using LinuxKit including a small test container which detects and uses all VM memory. We built the VM image by: $ git clone git://github.com/djs55/hyperkit-measure-memory $ cd hyperkit-measure-memory/cmd/touch $ make linuxkit-build We then booted the VM with 2GB of memory using hyperkit on a MacBook Pro running macOS 10.14 (Mojave): $ make linuxkit-run-hyperkit We then booted the VM with 2GB of memory using qemu on the same MacBook Pro also running macOS 10.14 (Mojave): $ make linuxkit-run-qemu Results The hyperkit process looked like this in Activity Monitor: Note the hyperkit process is using just under 4 GB of “Memory” but around 2 GB of “Real Mem”. The qemu process looked like this in Activity Monitor: Note the qemu process is also using just under 4 GB of “Memory” and also around 2 GB of “Real Mem”. Conclusions Since the qemu and hyperkit processes show the same double-accounting problem in Activity Monitor despite them being completely separate codebases, we conclude the bug must be in macOS itself, probably the Hypervisor.framework. We have reported the bug to Apple as bug number 48714544.
