Why highmem
Currently the 32 bit x86 architecture is the most popular type of computer. In this architecture, traditionally the Linux kernel has split the 4GB of virtual memory address space into 3GB for user programs and 1GB for the kernel.
On traditionally sized (for lack of a better euphemism) 32 bit systems, this meant the kernel could map all of physical memory into its address space, which makes it possible to:
- Directly access any memory from inside the kernel, which makes things a lot simpler and faster.
- Use pointers from kernel object to another. The alternative would be to track physical addresses, mapping in a physical address and calculating the virtual address of the structure before accessing it, and unmapping the physical address after use.
- Use huge pages from kernel space, resulting in more TLB coverage and better performance.
On other architectures, such as that of the MIPS processor, the hardware itself is only capable of directly accessing a portion of physical memory. A 32-bit MIPS processor normally runs its operating system in a 512 MiB portion of memory that has a direct mapping between the virtual and physical address. All other memory must be accessed through a translation lookaside buffer (TLB) and it is not capable of mapping all of the 4 GiB of physical memory into its address at the same time. Thus, its need for high memory support can be even more dire than that of the x86 architecture
Coping with highmem
However, many people insist on using more than 1GB of physical memory on such a 32 bit system. This makes it necessary for the Linux kernel to jump through some interesting hoops...
Basically the system uses the following tactics:
- Memory above the physical address of 896MB are temporarily mapped into kernel virtual memory whenever the kernel needs to access that memory.
Data which the kernel frequently needs to access is allocated in the lower 896MB of memory (ZONE_NORMAL) and can be immediately accessed by the kernel (see Temporary mapping).
- Data which the kernel only needs to access occasionally, including page cache, process memory and page tables, are preferentially allocated from ZONE_HIGHMEM.
- The system can have additional physical memory zones to deal with devices that can only perform DMA to a limited amount of physical memory, ZONE_DMA and ZONE_DMA32.
Allocations and pageout pressure on the various memory zones need to be balanced (see Memory Balancing).
Temporary mapping
The temporary mapping of data from highmem into kernel virtual memory is done using the functions kmap(), kunmap(), kmap_atomic() and kunmap_atomic().
The function kmap() gives you a persistant mapping, ie. one that will still be there after you schedule and/or move to another CPU. However, this kind of mapping is allocated under a global lock, which can be a bottleneck on SMP systems. The kmap() function is discouraged.
Good SMP scalability can be obtained by using kmap_atomic(), which is lockless. The reason kmap_atomic() can run without any locks is that the page is mapped to a fixed address which is private to the CPU on which you run. Of course, this means that you can not schedule between setting up such a mapping and using it, since another process running on the same CPU might also need the same address! This is the highmem mapping type used most in the 2.6 kernel.
Memory balancing
On a system with 2GB memory, a little more than half of the memory will be in high memory (ZONE_HIGHMEM) and a little less than half of memory will be in so-called low memory (ZONE_NORMAL and ZONE_DMA). It is important that the kernel allocates process and page cache memory from both zones and that the pageout code recycles pages from both zones, in proportion to each zone's size.
The reason for this is that applications might want to use all 2GB of memory. If the system ended up allocating and reclaiming high memory much faster than low memory, then the application would have part of its data swapped out from high memory, instead of resident in low memory. However, the better the system balances allocations and recycling between high and low memory, the closer the application's effective available memory size approaches the full 2GB.
Sysadmin considerations
The main danger of running on a 32 bit system with lots of highmem (more than 8GB) is that the kernel could end up needing to allocate more data than what fits in ZONE_NORMAL. This means the machine can effectively run out of memory, even if there were still lots of high memory free.
Another issue is that the system will more aggressively reclaim kernel data structures like cached inodes, buffer heads and other caches that can help system performance.
The third issue is that, on a 32 bit system, no process will be able to effectively use more than 3GB of memory. This means that buying more than 4GB of memory is only useful if none of the processes on your system need all of the memory.
For these reasons it is recommended that if you buy a system with more than 4GB of memory, you should consider getting a 64 bit CPU and installing a 64 bit operating system. The price difference between 32 and 64 bit systems is practically nonexistant in 2005, and 64 bit x86-64 systems can still run 32 bit applications, so there is no real need to experience the pains of highmem any more...
Update: in 2006, 64 bit systems are the same price as 32 bit systems. The only exception is laptops.