Currently the 32 bit x86 architecture is the most popular type of computer. In this architecture, traditionally the Linux kernel has split the 4GB of virtual memory into 3GB for user programs and 1GB for the kernel. However, many people insist on using more than 1GB of physical memory on such a 32 bit system. This makes it necessary for the Linux kernel to jump through some interesting hoops...
Basically the system uses the following tactics:
- Memory above the physical address of 896MB are temporarily mapped into kernel virtual memory whenever the kernel needs to access that memory.
- Data which the kernel frequently needs to access is allocated in the lower 896MB of memory (ZONE_NORMAL) and can be immediately accessed by the kernel.
- Data which the kernel only needs to access occasionally, including page cache, process memory and page tables, are preferentially allocated from ZONE_HIGHMEM.
- The system can have additional physical memory zones to deal with devices that can only perform DMA to a limited amount of physical memory, ZONE_DMA and ZONE_DMA32.
- Allocations and pageout pressure on the various memory zones are balanced.
The main danger of running on a 32 bit system with lots of highmem (more than 8GB) is that the kernel could end up needing to allocate more data than what fits in ZONE_NORMAL. This means the machine can effectively run out of memory, even if there were still lots of high memory free.
Another issue is that the system will more aggressively reclaim kernel data structures like cached inodes, buffer heads and other caches that can help system performance.
The third issue is that, on a 32 bit system, no process will be able to effectively use more than 3GB of memory. This means that buying more than 4GB of memory is only useful if none of the processes on your system need all of the memory.
For these reasons it is recommended that if you buy a system with more than 4GB of memory, you should consider getting a 64 bit CPU and installing a 64 bit operating system. The price difference between 32 and 64 bit systems is practically nonexistant in 2005, and 64 bit x86-64 systems can still run 32 bit applications, so there is no real need to experience the pains of highmem any more...
The temporary mapping of data from highmem into kernel virtual memory is done using the functions kmap(), kunmap(), kmap_atomic() and kunmap_atomic().
The function kmap() gives you a persistant mapping, ie. one that will still be there after you schedule and/or move to another CPU. However, this kind of mapping is allocated under a global lock, which can be a bottleneck on SMP systems. The kmap() function is discouraged.
Good SMP scalability can be obtained by using kmap_atomic(), which is lockless. The reason kmap_atomic() can run without any locks is that the page is mapped to a fixed address which is private to the CPU on which you run. Of course, this means that you can not schedule between setting up such a mapping and using it, since another process running on the same CPU might also need the same address! This is the highmem mapping type used most in the 2.6 kernel.
On a system with 2GB memory, a little more than half of the memory will be in high memory (ZONE_HIGHMEM) and a little less than half of memory will be in so-called low memory (ZONE_NORMAL and ZONE_DMA). It is important that the kernel allocates process and page cache memory from both zones and that the pageout code recycles pages from both zones, in proportion to each zone's size.
The reason for this is that applications might want to use all 2GB of memory. If the system ended up allocating and reclaiming high memory much faster than low memory, then the application would have part of its data swapped out from high memory, instead of resident in low memory. However, the better the system balances allocations and recycling between high and low memory, the closer the application's effective available memory size approaches the full 2GB.