Where did my memory go?
Users generally know precisely how much RAM is in a given system, usually because they _paid_ for that RAM. They want to be sure that all of it is put to good use and that none is wasted. However, the different memory reporting tools (top, /proc/meminfo, free, etc...) seem to be inconsistent with each other, and none seem to be able to account for _all_ of the memory installed in the system.
- How much memory did Linux get? How much was reserved by the hardware, firmware, or hypervisor? The baseline for the kernel is not the amount of memory in the system, but the amount handed to it by the hardware. I've seen cases where it was possible to add extra DIMMs to a system, the BIOS would recognize and show them, but they were completely unavailable to Linux.
- Fixed kernel memory reservations. These allocations will be the same for a given kernel image, and do not change with the amount of memory installed. They are not available for user allocations, and can not generally be "tuned". The only way to change their amounts is to change the kernel. Examples: kernel text and data structures
Memory: 24693684k/25952256k available (5484k kernel code, 787476k absent, 471096k reserved, 3204k data, 588k init)
- Variable memory reservations. These allocations will generally be the same on a given system, but can vary between systems, or with different kernel images. Their size depends on something external to the kernel like the number of CPUs in the system or the amount of memory.
- mem_map[]. This is a kernel structure used to manage and reclaim memory. It is also the largest _consumer_ of memory, and it grows larger as the amount of installed RAM increases. It uses ~1.36% of memory on 64-bit and 0.78% on 32-bit kernels. It is sometimes possible to decrease this amount by reconfiguring and recompiling your kernel to remove certain kernel features like kmemcheck, but it is not possible to change it if you are, for example, running a fixed distribution kernel.
- memory cgroup structures. This can be fixed by disabling cgroups at boot
- Kernel hash tables. Used to index kernel caches like dentry and inode caches. Larger caches generally mean higher performance and the kernel assumes that, as system RAM increases, it should devote more and more to these tables. The dentry hash is 32MB on a 24GB system, for example.
- Architecture-specific reservations:
- powerpc has a pagetable-equivalent structure referred to as the htab. It is required by hardware, and can be fairly large. Does it grow with memory size?
- Hypervisors will tend to eat lots of memory. Running a KVM guest, you'll pay that 1.36% for mem_map[] twice: once in the host, and once in the guest.