# Keeping gigabytes whole: the gigablock design
The gigablock series is a 36-patch change to the Linux page allocator, based on
`v7.3-rc1`. It concentrates non-movable pageblocks in a small number of
1 GiB-aligned ranges so that the remaining ranges can still supply gigantic
hugetlb pages after the machine has been running for a long time.
Patch numbers refer to the ordered series in section 10. Measurements from
older prototypes are labelled as such; they explain design choices but are not
presented as measurements of this exact revision.
The code is in the [riel/gigablock-2026-10-03](https://github.com/rikvanriel/linux/tree/riel/gigablock-2026-10-03) branch
## Contents
1. The problem
2. Scope, terms and activation
3. State and accounting
4. Allocation policy and supply
5. Evacuation
6. Bounds, liveness and edge cases
7. Measurements and validation
8. Limits and deferred work
9. Observability
10. Patch order
---
## 1. The problem
A 1 GiB gigantic page needs a 1 GiB-aligned physical range in which every page
is free or movable. One non-movable page can make the whole range unusable,
even if almost all of it is free.
The buddy allocator organizes free memory by order and migratetype. Free pages
in `MIGRATE_UNMOVABLE` and `MIGRATE_RECLAIMABLE` pageblocks normally serve
kernel allocations; free pages in `MIGRATE_MOVABLE` pageblocks serve movable
allocations. When the requested non-movable list has no sufficiently large
buddy, the fallback path reaches the movable lists and chooses a pageblock by
free-block size, without considering which 1 GiB range contains it.
Repeated fallbacks therefore spread a small amount of non-movable memory over
many gigantic-page-sized ranges. The problem is placement, not necessarily the
amount of non-movable memory. On a 765 GiB node after 53 days of build and
filesystem work, the median 1 GiB range was 13.0% non-movable while the machine
as a whole was 13.1% non-movable, and none of 754 ranges was clear of
non-movable pageblocks. On a 253 GiB `v7.3-rc1` test host, a metadata and memory
workload mixed 251 of 254 ranges in 9.2 hours and a request for eight 1 GiB pages
returned none.
The series changes the fallback decision. If a non-movable allocation must
cross into movable memory, it first looks for a usable movable pageblock in a
range that is already mixed. It also keeps a bounded reserve of free
non-movable pageblocks in those ranges, migrates movable content out of
non-movable pageblocks, and makes an unmixed range the last resort for callers
that can wait or retry smaller.
```mermaid
flowchart TD
A["non-movable allocation"] --> B{"own migratetype has a buddy<br/>of the requested order or larger?"}
B -->|yes| C["allocate inside an existing<br/>non-movable pageblock"]
B -->|no, stock behavior| D["claim the largest movable buddy<br/>without regard to 1 GiB location"]
D --> E["mix another gigablock"]
E --> F["repeat across the zone"]
F --> G["non-movable content becomes<br/>thinly spread across most gigablocks"]
B -->|no, gigablock policy| H["search already mixed gigablocks first"]
H -->|usable block| I["claim without losing another<br/>gigantic-page candidate"]
H -->|none| J["try other non-movable type,<br/>bounded evacuation, or caller fallback"]
J -->|last resort| K["mix an unmixed gigablock<br/>rather than fail the allocation"]
```
A prerequisite fix at the bottom of the series restores free-page accounting
for `CONFIG_DEBUG_PAGEALLOC` guard pages. When a guard buddy merges,
`clear_page_guard()` no longer updates free counters; the merged block otherwise
reaches a free list with `NR_FREE_PAGES` and `nr_free_nonmovable` short by the
guard pages. The fix credits the cleared guard pages under the merged block's
migratetype.
---
## 2. Scope, terms and activation
### Terms
| Term | Meaning | Code |
|---|---|---|
| pageblock | The migratetype unit; 512 pages at `pageblock_order == 9` on x86-64 | pageblock flags |
| gigablock | A physically aligned 1 GiB span; 512 pageblocks on x86-64 | `struct gigablock` |
| mixed gigablock | A tracked gigablock with at least one non-movable pageblock | `nr_nonmovable_blocks > 0` |
| unmixed gigablock | A tracked gigablock with no non-movable pageblock | `gigablock_is_unmixed()` |
| borrow | Movable content in a non-movable pageblock | `PB_migrate_mixed` |
| claim | A movable pageblock converted to the allocation's non-movable type | `__rmqueue_claim()` |
| target | The partly mixed gigablock tried first by claims | `zone->gigablock_target` |
| seed/refill | Whole free movable pageblocks converted ahead of demand | `gigablock_claim_ahead_blocks()` |
| release | Whole free non-movable pageblocks returned to movable lists | `nonmovable_release_excess()` |
| evacuation | Movable pages migrated out of a non-movable pageblock | `gigablock_evacuate_block()` |
| cycle | One traversal of the zone's gigablock index space by the shared evacuation cursor | `zone->gigablock_cycle` |
A gigablock is fixed at 1 GiB rather than `PUD_SIZE`. On arm64 with 16 KiB
pages a PUD is 64 GiB, while 1 GiB is a run of 32 PMDs that the TLB can
coalesce. Larger contiguous allocations require several aligned gigablocks.
### Build and boot gate
The feature is capable only when all three configuration options are enabled:
```c
CONFIG_CONTIG_ALLOC && CONFIG_HUGETLB_PAGE && CONFIG_ARCH_HAS_GIGANTIC_PAGE
```
`gigablocks=off` disables it at boot. A zone is tracked only when its
boot-time, 1 GiB-aligned span covers at least two gigablocks and the
per-gigablock metadata allocation succeeds. (A failed resistance-table
allocation only warns; the zone stays tracked.)
Kexec HandOver boots stay tracked. Preserved pages get initialized struct
pages and sit in memblock ranges the seed reads, so scans see them as
ordinary reserved memory holding non-movable content.
### Stable geometry and scan readiness
Metadata is allocated from the boot-time zone span. `gigablock_zone_start_pfn`
and `gigablock_zone_end_pfn` remain fixed even if hotplug later changes the
zone. Only pageblocks wholly inside that span, with both ends in the zone,
are tracked; a hole inside the pageblock does not exclude it. A partial
pageblock at either zone boundary is left outside tracking: it is never
counted, and the steal refusal and no-mix/last-resort logic do not cover
it. On arm64 with 64 KiB pages the untracked span can reach 512 MiB at
each boundary.
The early boot seed estimates presence and migratetype before all struct pages
exist. `page_alloc_init_late()` recounts each gigablock after deferred struct
page initialization and hugetlb bootmem initialization, then enables a static
key. Placement, seeding, release and evacuation do no page scans before that
point.
The fixed metadata does not try to follow memory hotplug. Before memory is
moved into or out of a tracked zone, `gigablock_hotplug_off()` permanently sets
`ZONE_GIGABLOCK_OFF`, clears the target and disables all placement, marker and
evacuation operations for that zone until reboot. Existing arrays remain
allocated so in-flight readers stay within their original bounds. Debugfs
reports the zone as disabled. This applies even if a hotplug operation later
rolls back.
---
## 3. State and accounting
```mermaid
classDiagram
direction LR
class zone {
nr_free_nonmovable
gigablocks[]
nr_gigablocks
gigablock_scan
gigablock_release_cursor
gigablock_target
claims_since_refill
nr_mixed_gigablocks
gigablock_evac_credits
gigablock_borrow_epoch
gigablock_swept_epoch
gigablock_epoch_probe_after
gigablock_epoch_probe_pending
gigablock_evac_cursor
gigablock_cycle
gigablock_cycle_epoch
gigablock_cycle_cleared
gigablock_evac_pass
gigablock_evac_block
gigablock_evac_lock
}
class gigablock {
nr_nonmovable_blocks
nr_nonmovable_blocks_seen
total_blocks
scan_block
scan_missed
resisted[]
}
class pageblock_bits {
migratetype
PB_migrate_mixed
PB_has_nonmovable
}
class task_struct {
in_gigablock_putback
}
class pglist_data {
kcompactd_evacuate_zones
}
zone "1" --> "many" gigablock : boot-time 1 GiB spans
gigablock "1" --> "many" pageblock_bits : valid pageblocks
pglist_data "1" --> "many" zone : kcompactd bitmap
```
### Per-pageblock state
The series adds two pageblock bits:
```c
PB_migrate_mixed /* non-movable block may contain movable pages */
PB_has_nonmovable /* last content scan found non-movable pages */
```
`PB_migrate_mixed` is a conservative hint. It is set when a claim leaves
movable pages behind, when a movable allocation takes from a non-movable free
list, when highatomic conversion can strand movable content, when
`__isolate_free_page()` keeps a block non-movable, when compaction capture puts
a movable request in a non-movable block, and when an LRU arrival is observed
in a non-movable block. An evacuation scan repairs a missing hint. The bit
may remain set after the movable pages leave; that costs a scan but does not
hide work.
The bit is cleared when the pageblock becomes movable, when a whole-block
buddy is added to a free list, or when compaction captures a whole block,
which bypasses the free list. Moving a pageblock to a movable migratetype and
clearing the bit use one masked pageblock-flags update, so a racing arrival
cannot leave the type and marker in an impossible intermediate state. The
mixed-marker mutators are no-ops for an untracked or hotplug-disabled zone.
There is deliberately no aggregate `nr_borrowed_blocks`. Maintaining an exact
level across lockless LRU arrivals and type changes introduced a race between
the bit and the aggregate. Liveness is carried by the borrow and swept epochs;
the pass always scans both marked and unmarked non-movable pageblocks.
`PB_has_nonmovable` is a last-observed content snapshot. Evacuation sets or
clears it after scanning a non-movable pageblock, and whole-block free clears
it. It can miss an unscanned block and can retain a stale positive until the
next scan. Placement does not use it; diagnostics print the per-gigablock
aggregate derived from it.
Hugetlb folios are counted separately. They are movable for contiguous-range
analysis, but gigablock evacuation has no hugetlb isolation path, so they are
excluded from the actionable movable count. The post-migration recount still
sees them, which prevents a block containing hugetlb pages from being reported
as cleared.
### Per-gigablock state
```c
struct gigablock {
unsigned int nr_nonmovable_blocks;
unsigned int nr_nonmovable_blocks_seen;
unsigned int total_blocks;
unsigned int scan_block;
unsigned int scan_missed;
u32 *resisted;
};
```
`nr_nonmovable_blocks` is maintained at the pageblock type-change site under
`zone->lock`. The first non-movable pageblock increments
`zone->nr_mixed_gigablocks`; removal of the last decrements it.
`total_blocks` contains only pageblocks the common validity predicate can walk
whole. `scan_block` resumes the next mixed-gigablock claim inside this
range. `scan_missed` accumulates misses across bounded scans and records a full
sweep only after the cursor has passed every pageblock since the last hit.
`resisted[]` is one `u32` per pageblock when `CONFIG_COMPACTION` is enabled.
The low 16 bits store the actionable movable-page count, the high 16 bits the
evacuation cycle, zero means no record, and `U32_MAX` means armed but not
recorded. A matching record suppresses attempts in its recording cycle and
the next three, so a permanently unchanged block is retried once every four
cycles. A new allocator mark, LRU arrival, type change or whole-block free
clears it.
### Per-zone and per-task state
The main zone fields are:
| Field | Purpose |
|---|---|
| `nr_free_nonmovable` | Exact free pages on unmovable and reclaimable buddy lists; shown in `/proc/zoneinfo` |
| `nr_mixed_gigablocks` | Number of tracked gigablocks with a non-movable pageblock |
| `gigablock_target` | Preferred partly mixed gigablock, or `ULONG_MAX` |
| `gigablock_scan` | Round-robin claim-search cursor |
| `gigablock_release_cursor` | Round-robin release-selection cursor |
| `claims_since_refill` | Smaller-only claims since the last refill/release attempt |
| `gigablock_evac_credits` | Saturating count of requested blocks, capped at 64 |
| `gigablock_borrow_epoch` | Lockless movable-arrival generation |
| `gigablock_swept_epoch` | Start epoch of the last complete empty cycle |
| `gigablock_epoch_probe_after` | Earliest time an epoch-only rejection may be probed |
| `gigablock_epoch_probe_pending` | A claim authorized one epoch-bypass pass |
| `gigablock_evac_cursor` | Next gigablock for the shared evacuation walk |
| `gigablock_cycle`, `_cycle_epoch`, `_cycle_cleared` | Full-cycle liveness state |
| `gigablock_evac_pass`, `_evac_block` | Marked/unmarked pass and first unchecked block within the current gigablock |
| `gigablock_evac_lock` | Serializes kcompactd and debugfs passes |
`task_struct->in_gigablock_putback` prevents an evacuator's own failed-folio
putbacks from looking like new arrivals.
`pgdat->kcompactd_evacuate_zones` is the per-node bitmap of zones with requested
or autonomous evacuation work. There is no per-node shrink state or work item.
---
## 4. Allocation policy and supply
### 4.1 Closing unaccounted crossings
```mermaid
stateDiagram-v2
direction LR
[*] --> Movable
Movable --> NonmovableWhole : whole-run claim, seed, or refill
Movable --> NonmovableBorrowed : partly used claim; set mixed hint
NonmovableWhole --> NonmovableBorrowed : movable allocation or LRU arrival
NonmovableBorrowed --> NonmovableWhole : evacuation clears movable content; hint may remain
NonmovableWhole --> Movable : release or checked movable claim
NonmovableBorrowed --> NonmovableBorrowed : movable conversion refused while kernel content remains
Movable --> Movable : non-movable steal or capture refused
```
The accounting is useful only if every non-movable use of a tracked movable
pageblock reaches a type change or is refused. The series closes three paths:
1. `__rmqueue_steal()` does not let a non-movable allocation take individual
pages from a tracked movable pageblock while leaving its type unchanged.
2. A non-movable claim converts every pageblock in the free run it takes. A
partly used pageblock is converted and marked `PB_migrate_mixed`.
3. `compaction_capture()` refuses a tracked movable block to a non-movable
request. The block reaches the movable free list and the allocation retry
goes through the normal claim path, which either converts and counts it or
refuses it. Other captures update the mixed marker consistently.
The explicit movable-conversion paths inspect content first.
`__isolate_free_page()` keeps a failing block non-movable, marks it mixed and
counts `gigablock_keep_nonmovable`. Highatomic unreserve keeps such a block
reserved on its first pass; later passes, including the forced pass before OOM,
release it as `MIGRATE_UNMOVABLE`.
### 4.2 Mixed-gigablock claim search
```mermaid
flowchart TD
A["non-movable allocation"] --> O{"own free list serves<br/>requested order?"}
O -->|yes| DONE["allocate"]
O -->|no| WALK["walk orders from MAX_PAGE_ORDER down;<br/>other non-movable type precedes movable"]
WALK --> F{"fallback selected<br/>at this order?"}
F -->|other non-movable type| CLAIM["claim if compatible"]
CLAIM -->|served| DONE
CLAIM -->|not claimable| MORE
F -->|first movable fallback| Q{"request below pageblock_order<br/>and scans ready?"}
Q -->|yes| M["try target, then round-robin<br/>eligible mixed gigablocks<br/>4 content / 256 block / 64 region limits"]
M -->|buddy of requested order or larger| MC["claim whole run or one used block;<br/>mark movable content"]
MC --> DONE
M -->|none| EV["request evacuation if gate opens"]
EV --> C
Q -->|no| C{"movable candidate is in<br/>an unmixed tracked gigablock?"}
C -->|no| CLAIM
C -->|HAS_FALLBACK| REF["refuse candidate"]
C -->|otherwise, first pass| SAVE["remember skipped candidate"]
REF --> MORE{"more orders?"}
SAVE --> MORE
F -->|none| MORE
MORE -->|yes| WALK
MORE -->|no| NF{"ALLOC_NOFRAGMENT?"}
NF -->|no| STEAL["try steal fallbacks;<br/>tracked movable blocks are refused,<br/>untracked movable blocks remain eligible"]
NF -->|yes| AFTER
STEAL -->|served| DONE
STEAL -->|none| AFTER{"ordinary skipped<br/>candidate remembered?"}
AFTER -->|yes| LAST["last-resort claim without<br/>repeating mixed search"]
LAST -->|served| SEED["claim unmixed block; seed if headroom permits"]
LAST -->|not claimable| RET
SEED --> DONE
AFTER -->|no| RET["return NULL;<br/>caller falls back or tries next zone"]
```
For requests below `pageblock_order`, the first movable fallback causes one
mixed-gigablock search:
1. Try the target first, with half of the four content scans and half of the 256
pageblock checks reserved for the rest of the zone.
2. Continue round-robin from `zone->gigablock_scan` through other eligible mixed
gigablocks.
3. Accept a movable pageblock only if it contains a buddy of the requested order
or larger. A sum of smaller buddies cannot serve the allocation on this
path.
4. Take a buddy of `pageblock_order` or larger whole, retyping every pageblock in
the run. Otherwise convert the one pageblock, move its free pages to the
requested non-movable type, mark any movable content, and allocate from it.
5. If the search misses, continue the ordinary size-ordered fallback. The
series never fails an allocation merely because placement could not help.
Each claim has three independent limits:
| Limit | Value | What it charges |
|---|---:|---|
| content scans | 4 | movable pageblocks whose pages are inspected |
| pageblock checks | 256 | validity and migratetype checks |
| gigablock visits | 64 | eligible, partly mixed gigablocks offered to the scanner |
Unmixed, empty and fully non-movable descriptors do not consume the 64 eligible
visits. Charging them would create a false 64 GiB placement horizon on a large
zone and could mix a clean gigablock while a usable mixed one existed later.
The descriptor walk can therefore cover the zone; the separate pageblock and
content limits bound the expensive work. An exact-tip function-graph run under
claim load recorded a 6.329-microsecond maximum in
`gigablock_alloc_flags()`. It did not attribute the mixed search itself, and
that build had no IRQ-off tracer, so very large-zone IRQ-off cost remains a
measurement concern.
The target is chosen by a full scan only during boot and the late recount. At
runtime no target replacement scans the zone: an empty target is invalidated by
the type transition, a full target is invalidated when its next claim misses,
and the next successful round-robin claim elects its gigablock. A type change
that creates a partly mixed gigablock also elects it when no target exists. A
fragmented, partly mixed target can still miss repeatedly, but it receives only
half of the per-claim pageblock and content budgets.
### 4.3 Refusing an unmixed gigablock
One allocation class can postpone the last-resort crossing:
`ALLOC_HAS_FALLBACK`, a higher-order `__GFP_NORETRY` allocation with a
smaller caller-level fallback. It skips the unmixed block independent of
the gate, because the smaller retry can use existing non-movable free
memory.
At each order the other non-movable type is tried before movable. If all
safer choices fail, an allocation without a fallback still takes the
unmixed gigablock; preservation must not turn into allocation failure.
Bulk refills set no flag, so only the universal last-resort rule steers
them; a refill that enters steal mode without a claim ends short.
### 4.4 Seed, refill and release
```mermaid
flowchart TD
MIX["claim first mixes a gigablock"] --> HS{"watermark and one-third<br/>share leave headroom?"}
HS -->|yes| SEED["seed wholly free movable runs"]
HS -->|no| HOLD1["do not seed"]
CLAIMS["64 qualifying smaller-only claims"] --> HR{"watermark and one-third<br/>share leave headroom?"}
HR -->|yes, target exists| FILL["refill wholly free runs in target"]
HR -->|yes, no target| HOLD2["wait for a target"]
HR -->|no| EXCESS{"pool is above high watermark?"}
EXCESS -->|yes| RELEASE["choose one of at most 16 round-robin candidates;<br/>release up to 32 blocks from it"]
EXCESS -->|no| HOLD3["nothing to release"]
EVAC["evacuation clears movable content<br/>from a non-movable block"] --> POOL["non-movable free lists<br/>nr_free_nonmovable"]
SEED --> POOL
FILL --> POOL
POOL --> ALLOC["future non-movable allocations<br/>avoid another crossing"]
RELEASE --> MOV["movable free lists"]
```
A claim that first mixes a gigablock may seed it with wholly free movable
pageblocks. Every 64th qualifying smaller-only claim may refill the target.
Both are limited by one snapshot of two headroom values:
- free non-movable pages stay below the zone's high watermark;
- free non-movable pages stay below one third of all free pages;
- no call intentionally takes more than 512 pageblocks, though the final buddy
run is moved whole and can cross that count.
Whole-run conversion matters. Moving the free pages of a multi-pageblock buddy
without changing every covered pageblock's type would put movable-typed blocks
on a non-movable free list and break both accounting and fallback behavior.
Free pages later accumulate on the non-movable lists. When the pool is above
the high watermark, `nonmovable_release_excess()` returns wholly free
unmovable or reclaimable blocks to movable lists. It excludes the target,
looks at at most 16 candidate gigablocks from a round-robin cursor, and returns
at most 32 pageblocks per call, with the last whole buddy run allowed to cross
the count. It records both a selected gigablock with no releasable block and a
16-entry window with no candidate. Release is called from the 64-claim refill
interval even when no target exists, so a barren candidate cannot permanently
block progress elsewhere.
This series does not invoke slab shrinkers. Shrinking the slab
preemptively may help prevent spilling into more gigablocks, but
the first version of the series should stay smaller.
---
## 5. Evacuation
In order to concentrate kernel allocations in a small number of gigablocks,
it is necessary to free up pages in those gigablocks that were used by
movable allocations. The current version of the series uses kcompactd
to evacuate some page blocks inside gigablocks that have both movable and
non-movable allocations.
### 5.1 Requests and the gate
```mermaid
flowchart TD
A1["mixed search claims a partly used block"] --> G
A2["mixed search finds no block"] --> G{"claim-time evacuation gate"}
G -->|non-movable free at or above high| W["gate_wmark; no request"]
G -->|non-movable share at or above one third| S["gate_share; no request"]
G -->|swept epoch equals borrow epoch| E{"32-second claim probe due?"}
E -->|no| EC["gate_epoch; no request"]
E -->|yes| P["gate_probe; authorize one pass"]
G -->|open| H{"zone still has a mixed gigablock?"}
P --> H
H -->|no| EMPTY["queue helper does nothing"]
H -->|yes| QW["add saturating credit and zone bit"]
QW --> WAKE{"rmqueue or bulk exit<br/>permits a wake?"}
WAKE -->|yes, after zone lock drops| K["kcompactd admission"]
WAKE -->|no| POLL["later wake or node poll"]
POLL --> K
K --> U{"watermark, share and epoch<br/>or pending probe allow work?"}
U -->|no| KEEP["keep credits for a later pass"]
U -->|yes| TAKE["take credits; request max of credits<br/>and high-watermark shortfall"]
TAKE --> RUN["bounded evacuation pass"]
RUN --> REFUND["refund credits not satisfied"]
```
Two events request evacuation, each only when the claim-time gate passes: a
partly used claim made by the mixed-gigablock search, and a mixed-search miss.
Size-ordered claims mark the block but do not request a pass. A request adds
one credit with a saturating compare/exchange, up to 64, sets the zone bit in `kcompactd_evacuate_zones`, and defers the actual wake until after
`zone->lock` is released. `rmqueue()` wakes only for allocations carrying
`ALLOC_KSWAPD`, and bulk exit only for `__GFP_KSWAPD_RECLAIM`.
`ALLOC_HAS_FALLBACK` requests therefore wait for a later wake-capable allocation
or the kcompactd poll.
The gate evaluates these tests in order:
1. **watermark:** stop if `nr_free_nonmovable` already meets the zone high
watermark;
2. **share:** stop if the non-movable lists hold at least one third of all free
pages;
3. **epoch:** stop if a complete cycle cleared nothing and no observed movable
arrival advanced the epoch since that cycle began.
There is no `nr_borrowed_blocks` or “nothing” gate. The mixed marker is a hint,
and pass 1 deliberately scans unmarked non-movable pageblocks, so an aggregate
of markers was neither necessary nor race-free.
The epoch gate has a bounded escape. A claim rejected only by the epoch test
starts a 32-second deadline. A later claim after the deadline records
`gigablock_evac_gate_probe` and authorizes one pass. The probe bypasses only the
epoch test: watermark and share are evaluated first. kcompactd polling honors
a pending probe but cannot create one by itself.
Wake-capable requests wake kcompactd immediately. Tracked nodes also poll at
500 milliseconds with proactive compaction enabled, backing off to 32 seconds
when a proactive pass makes no progress; with proactive compaction disabled the
poll is 32 seconds. A closed gate leaves the credits in place for a later poll
or wake. For each zone whose gate is open, kcompactd serves the larger of queued credits
and the pageblock shortfall below the high watermark, capped at 64. The gate
is checked before credits are consumed. Credits a bounded pass did not satisfy
are returned through the same saturating helper.
### 5.2 One pass
```mermaid
flowchart TD
START["hold gigablock_evac_lock;<br/>resume shared gigablock, phase, and block cursors"] --> STOP{"one cycle or visit limit, requested clears,<br/>fatal signal, zone disabled, or -ENOMEM?"}
STOP -->|yes| SAVE["save first unchecked block<br/>or next unvisited gigablock"]
STOP -->|no| MIX{"gigablock is mixed<br/>and has valid pageblocks?"}
MIX -->|no| NEXT["advance gigablock cursor"]
MIX -->|yes| PHASE{"saved phase"}
PHASE -->|0| MARKED["scan marked non-movable blocks"]
PHASE -->|1| UNMARKED["scan unmarked non-movable blocks"]
MARKED --> CAND["check one pageblock in one IRQ-off section"]
UNMARKED --> CAND
CAND --> LIMIT{"check or attempt limit reached?"}
LIMIT -->|yes| SAVE
LIMIT -->|no| COUNT["count kernel, reserved, and actionable<br/>movable pages net of hugetlb"]
COUNT -->|unchanged resistance record| SKIP["skip until record expires"]
COUNT -->|no actionable movable pages| STALE["count stale"]
COUNT -->|actionable pages| MIG["repair hint; arm record;<br/>drain LRU batching; migrate"]
MIG --> RECOUNT["recount with hugetlb included"]
RECOUNT -->|no movable page remains| CLEAR["clear one requested block"]
RECOUNT -->|movable pages remain| STUCK["record stuck reason;<br/>an error reports aborted"]
STUCK --> FATAL{"migration error is -ENOMEM?"}
FATAL -->|yes| SAVE
FATAL -->|no| MORE
SKIP --> MORE{"more blocks in this phase?"}
STALE --> MORE
CLEAR --> MORE
MORE -->|yes| CAND
MORE -->|no, phase 0| SWITCH["switch to unmarked phase"]
SWITCH --> UNMARKED
MORE -->|no, phase 1| NEXT
NEXT --> STOP
```
The pass walks mixed gigablocks from a shared cursor. Within each gigablock it
scans non-movable pageblocks in two phases: blocks marked
`PB_migrate_mixed`, then unmarked blocks. It checks one pageblock in each
IRQ-off section and reschedules between checks.
For a candidate block it:
1. counts non-movable, reserved and actionable movable pages, hugetlb
netted out, under `zone->lock` and refreshes `PB_has_nonmovable`;
2. skips an unchanged resistance record;
3. repairs a missing mixed marker;
4. drains LRU batching lazily, only before the first migration attempt;
5. isolates and migrates movable folios without `zone->lock`;
6. puts failed folios back after each batch while suppressing false arrival
accounting;
7. recounts the block and calls it cleared when no movable page remains,
hugetlb included; otherwise it records the stuck reason.
Passes use `MIGRATE_SYNC_LIGHT`, disable LRU caches globally from the first
migration attempt until the pass ends, and may record resistance.
The series does not empty and claim movable pageblocks as an evacuation
fallback. That recovery experiment was dropped because it migrates from a
block that remains movable while migration runs, so destination allocations can
reuse the just-freed source pages. Its aggregate free-page precheck guaranteed
neither a destination of the needed order nor a coalesced pageblock buddy.
### 5.3 Bounds and resume state
A kcompactd or debugfs pass tries migration in at most 64 blocks, checks at
most 256 pageblocks and visits at most 64 gigablocks.
A failed or stale candidate consumes a check but not a requested clear. The
attempt bound limits expensive migration even when nothing clears; the check
and visit bounds keep LRU batching and the shared mutex from spanning a zone.
If a bound, signal or `-ENOMEM` stops the pass inside a gigablock, the saved
phase and pageblock point to the first unchecked work. If the gigablock is
finished, the gigablock cursor points to the next unvisited one. This avoids
restarting at a resistant prefix.
### 5.4 Resistance and cycles
```mermaid
stateDiagram-v2
[*] --> None
None --> Armed : pass before migration
Armed --> Recorded : cmpxchg stores actionable count and cycle
Armed --> None : block cleared or concurrent arrival clears slot
Armed --> Armed : migration error stores nothing
Recorded --> None : new mark, LRU arrival, type change, or whole-block free
Recorded --> Recorded : same count in recording cycle plus next three
Recorded --> Armed : nonzero count changed or record expired
```
Before a migration the pass writes `GIGABLOCK_RESIST_ARMED`. If no
arrival clears it while the lock is dropped, the pass stores the actionable
movable-page count and cycle with `cmpxchg()`. A matching record suppresses
another attempt in its recording cycle and the next three; the block is
retried once every four cycles. Different content with the same count cannot
hide behind the record because arrivals and type changes clear it first. A
migration error leaves the armed value, which matches no count.
The unbounded cursor value identifies the cycle as
`cursor / nr_gigablocks`. A cycle records its starting borrow epoch.
```mermaid
sequenceDiagram
participant E as evacuation cursor
participant Z as zone cycle state
participant A as movable arrivals
E->>Z: enter cycle, cycle_epoch = borrow_epoch, cycle_cleared = false
A->>Z: new mixed mark increments borrow_epoch
A->>Z: first arrival in a cycle advances epoch with cmpxchg
E->>Z: clear a block, cycle_cleared = true
E->>Z: cross into next cycle
alt finished cycle cleared nothing
E->>Z: swept_epoch = prior cycle_epoch
alt borrow_epoch still equals swept_epoch
Note over Z: epoch gate closes
else an arrival advanced borrow_epoch
Note over Z: epoch gate remains open
end
else finished cycle cleared work
Note over Z: epoch gate remains open
end
```
When a cycle finishes without clearing a block, it copies the cycle's start
epoch to `gigablock_swept_epoch`. If no arrival advanced
`gigablock_borrow_epoch`, the equal values close the epoch gate. A movable
arrival advances the borrow epoch once per cycle, so an arrival the cursor
already passed still keeps the gate open for another traversal.
---
## 6. Bounds, liveness and edge cases
### Lock and latency bounds
- The mixed claim search reads at most four pageblocks and checks at most 256.
Its 64-region budget charges only eligible mixed gigablocks; descriptor-only
skips remain a large-zone measurement concern.
- Seed/refill and release hold `zone->lock`, but operate on one gigablock at a
time and have explicit block budgets. Release candidate selection looks at
16 gigablocks, not the whole zone.
- Runtime target replacement never scans the zone.
- Evacuation checks one pageblock per IRQ-off section and has attempt, check and
visit limits.
- `extfrag/gigablock_contig` uses a private seq iterator and emits one
gigablock per record, so a small userspace read buffer does not rescan the
same gigablock's pages.
### Allocation safety
- Skipping an unmixed gigablock never fails an allocation by itself. A
fallback caller retries at a smaller order; every other allocation takes
the block on the last-resort claim.
- `gigablocks=off`, an incapable build and a hotplug-disabled zone make mixed
marker mutators and all policy entry points no-ops.
### Hotplug, holes and boundaries
The boot-time metadata is not resized. Hotplug disables the affected tracked
zone before its span changes, and all type updates, markers, claims, refill,
release, evacuation and diagnostics honor that state. The gigablock listing
reports “tracking disabled after memory hotplug”; the contig reader skips
the disabled zone.
The common validity rule requires both ends of a pageblock to be in the same
zone and online before runtime scans touch it. Holes and partial boundary
blocks remain outside the counts. This costs capacity at boundaries but avoids
walking uninitialized or foreign struct pages.
### Gigantic hugetlb pages
Evacuation's compaction control uses `whole_block_folio` rather than
`alloc_contig`, so it can isolate an ordinary pageblock-sized folio without
entering the hugetlb dissolve path. Hugetlb folios have no gigablock evacuation
path. They are excluded from actionable movable counts and resistance
fingerprints, but retained in the final recount. A block that also held
actionable pages can be reported as stuck reason `hugetlb`; a block whose only
movable count is hugetlb is stale and is not migrated. `gigablock_contig`
reports `huge` only for a folio of at least 1 GiB. On architectures whose
hstate supports migration, smaller hugetlb folios pass its unmovable-page test
and are not blockers. `alloc_contig_range()` behavior is unchanged.
### Disabled automatic slab shrink
An automatic global/root slab pass was implemented and tested, but not retained.
On matched 256 GiB hosts creating 39.6 million empty files, the treatment ran
3,120 passes, freed 392 objects in 316,812 microseconds, and did not reduce slab,
mixed-gigablock or non-movable-pageblock growth. The final series contains no
shrink work item, memcg iteration, shrink state, vmstat events or tracepoint.
Slab remains ordinary reclaim's responsibility.
---
## 7. Measurements and validation
### Historical placement soak
Two identical single-node 256 GiB Bergamo machines ran an earlier
prototype for 22.1 hours under anonymous memory, tmpfs, 64 GiB of page cache and
a slow stream of dentries, sockets and page tables. That prototype differed
materially from the final series: claim admission used total free pages rather
than a buddy of the requested order, frozen kernel pages were misclassified,
failed migration folios were retried within a block, the evacuator's own
putbacks looked like arrivals, and pass bounds and target handling were older.
| Result | Host A | Host B |
|---|---:|---:|
| mixed gigablocks | 33 of 254 | 32 of 254 |
| 1 GiB pages from a request for eight | 8 in 40 ms | 8 in 441 ms |
| mixed-gigablock claims / target-served | 1,036 / 1,030 | 788 / 783 |
| pageblocks cleared by evacuation | 9,076 | 9,053 |
| pages migrated | 583 million | 570 million |
| direct passes | 0 | 0 |
Stock `v7.3-rc1` on the same class of hardware and workload mixed 251 of 254
gigablocks in 9.2 hours and supplied none of eight requested 1 GiB pages. A
matched 17-hour run mixed 252 of 254 and supplied 15 of 100 requests. These
results establish the placement effect, not the exact cost or behavior of the
final implementation.
### Bulk wake test
A 250 GiB test host running a recent version of the code made 4,096 calls
to `alloc_pages_bulk(GFP_NOWAIT | __GFP_NOWARN)` for 512 pages each.
`GFP_NOWAIT` carries `__GFP_KSWAPD_RECLAIM`, which the later wake condition
requires. Of those, 312 calls queued evacuation work. Every one observed a
wake before the bulk call returned; first request to first wake was at most 143
microseconds and 36.9 microseconds on average. Afterwards the host allocated
and released eight 1 GiB pages without a warning. This predates the final
fold but exercises the bulk wake mechanism retained in it.
### Final revision validation
The final validation covered:
- every exact final tree built per commit for `CONFIG_COMPACTION=y`,
`CONFIG_COMPACTION=n`, and `CONFIG_COMPACTION=n` with CMA enabled;
- final bzImage builds for those configurations and `CONFIG_MEMCG=n`;
- checkpatch with no errors (two quoted-string-split warnings in one
zoneinfo hunk); changelog, static and function-growth checks;
- exact-tip warning-free VM ramfs, churn, wake, contiguous-allocation,
baseline, hugetlb and duration-control workloads; the exact-tip config had
lockdep and `DEBUG_VM` disabled;
- a separate `CONFIG_ACPI_HOTPLUG_MEMORY=y` kernel with
`memhp_default_state=offline`: 2 GiB add, explicit online into tracked Normal,
permanent tracking disable, offline, device removal, seq output,
evacuation and contig reads. The default auto-online run placed the DIMM in
untracked `ZONE_MOVABLE` and correctly did not exercise the disable path.
QMP removed the DIMM device in both runs, while immediate `object-del` of its
backend returned “in use”;
- exact-tip bare-metal boot, metadata walk, evacuation request and four 1 GiB
hugepage allocations with no kernel warning. The evacuation request
scanned 155 pageblocks and cleared none.
- an exact-tip KHO round-trip in a 16 GiB VM (`test_kho`, 1 GiB preserved,
`PROVE_LOCKING` + `DEBUG_VM`): first boot tracked, kexec into a KHO boot
still tracked, `KHO restore succeeded`, a debugfs evacuation write ran
clean, and the serial log had no warning, lockdep report or oops.
The ENOMEM harness completed, but the build lacked `fail_page_alloc`; its four
warning matches were intentional SysRq-L traces. Exact-tip function-graph
maxima were 45.298 microseconds for background evacuation and 6.329
microseconds during claim load in `gigablock_alloc_flags()`. Wall-clock
time was 0.56 milliseconds for the evacuation write. The build had no
IRQ-off tracer, so these are function and wall-time measurements rather than
IRQ-off bounds. The VM, hotplug and bare-metal runs above used the 39-patch
pre-drop tip `c3b15041d6c5`; no direct pass ran in any of them, and the
per-commit and bzImage builds were re-run on this exact tip.
---
## 8. Limits and deferred work
### Accepted limits
These are behaviors the series ships with, documented so reviewers know
what not to expect.
- The current test evidence does not include injected `fail_page_alloc`
ENOMEM.
- Skipping an unmixed gigablock can move the allocation to the next zone in
the zonelist, a lower local zone or a remote node, before the caller falls
back or the last-resort claim runs. That ordering has no locality
preference.
- Movable-typed pageblocks are never evacuation candidates. A non-movable
allocation that finds a free page in one converts the whole block, marks
it mixed, and evacuation then moves the movable strays out — but a block
that is 100% full of page cache never enters that chain until reclaim
frees something in it. Migrating a movable block's contents out without
needing a free page first (the shelved recovery experiment) would close
that gap.
- Orders at or above `pageblock_order` use the size-ordered claim path rather
than the mixed-gigablock search.
- Highatomic reservation and cross-type buddy adoption can still choose a
gigablock without placement policy. Their type changes remain accounted.
- A movable page allocated with `__GFP_MOVABLE` that never reaches the LRU or
registers `movable_ops` can remain non-movable content in a movable block,
with nothing identifying the producer.
- Resistance is a count fingerprint, not a classification of every migration
failure. A permanently stuck block is retried once every four cycles.
- Release is driven by the 64-claim refill interval. A zone with no qualifying
claims does not run release autonomously.
- Dynamic hotplug metadata is not implemented. A span change disables the
affected zone until reboot.
- Partial boundary pageblocks are outside tracking, counting and placement
protection; the untracked span can reach 512 MiB at each boundary on arm64
with 64 KiB pages.
### Placement and search follow-ups
- Untracked edge and hole-adjacent blocks serve kernel demand only through
the normal fallback order. Trying them first, since they can never form
part of a gigantic page, remains future work.
- The mixed-gigablock claim search scans in the allocation hot path under
`zone->lock` with interrupts off. Moving that work into
watermark-maintenance slow paths, so claims need no hot-path scan, remains
future work. Reducing claim-ahead overhead in the fast path is part of
the same effort.
- A fragmented, partly mixed target can consume its half of the search budget
on repeated misses. Runtime replacement is bounded and event-driven, not an
order-specific target cache; whether an order-aware cache pays for itself is
unmeasured.
- The eligible-only 64-gigablock budget does not bound descriptor reads across a
very large zone. The expensive content and pageblock work is bounded; the
descriptor-only worst case remains a measurement gate.
- `nonmovable_release_excess()` picks a gigablock by typed-block count,
which says nothing about whether it holds wholly free blocks, then scans
it. Walking the non-movable free lists at `pageblock_order` and above
instead, returning whole free runs until back under the threshold with no
gigablock selection, remains future work.
### Evacuation follow-ups
- The evacuation pass and its helpers live in `mm/page_alloc.c`. Cleaning
them up and moving them to a better home remains future work.
- Evacuation requests name a zone, and each pass scans from the zone cursor.
Requesting evacuation for a particular gigablock, so passes go straight
to the work, remains future work. Once evacuation and compaction are more
targeted, the compaction epoch can be replaced with something smarter.
- Background evacuation reuses kcompactd, and the request, wake, gate and
credit plumbing around it is larger than the earlier standalone evacuator.
Two simplifications are on the table: separate evacuation code instead of
reusing kcompactd, or async evacuation from a work item on the dropped
direct-evacuation shape, either of which would shed most of the plumbing.
Direct evacuation itself was dropped before the first posting for lack of
test coverage.
### Series-order and naming follow-ups
- Move "mm/page_alloc: track pageblocks with non-movable pages seen by
scans" and "mm/vmstat: report non-movable pageblocks per gigablock"
earlier in the series, so reviewers see before-and-after results with
the other allocator changes.
- Go over all the code and comments again, to make sure terminology
is consistent with the rest of mm/
- Extract the fixes that can go upstream indendently of this series.
---
## 9. Observability
### `/proc/vmstat`
| Area | Counters |
|---|---|
| crossings | `gigablock_claim_free_block`, `gigablock_claim_used_block`, `gigablock_movable_claim_free`, `gigablock_movable_claim_used`, `gigablock_convert_to_nonmovable`, `gigablock_convert_to_movable` |
| claim reason | `gigablock_claim_smaller_only`, `gigablock_claim_nothing_free` |
| mixed search | `gigablock_claim_from_mixed`, `gigablock_claim_full_scan`, `gigablock_claim_partial_scan`, `gigablock_claim_scan_limit`, `gigablock_no_movable_blocks` |
| target and supply | `gigablock_target_tried`, `gigablock_target_served`, `gigablock_target_moved`, `gigablock_claim_ahead_blocks`, `gigablock_claim_ahead_wmark`, `gigablock_claim_ahead_clamped`, `gigablock_refill` |
| release | `gigablock_release_block`, `gigablock_release_empty`, `gigablock_release_no_candidate` |
| unmixed refusal | `gigablock_mix_fallback`, `gigablock_mix_avoided` |
| gate | `gigablock_evac_gate_open`, `gigablock_evac_gate_epoch`, `gigablock_evac_gate_probe`, `gigablock_evac_gate_wmark`, `gigablock_evac_gate_share` |
| cycles and requests | `gigablock_evac_epoch_arrival`, `gigablock_evac_cycle_empty`, `gigablock_evac_request` |
| evacuation | `gigablock_evac_scanned`, `gigablock_evac_cleared`, `gigablock_evac_migrated` |
| refused conversions and arrivals | `gigablock_steal_skipped`, `gigablock_keep_nonmovable`, `gigablock_movable_added_marked`, `gigablock_movable_added_unmarked`, `highatomic_free_to_movable` |
| contiguous search | `contig_frozen_range_skipped`, `contig_frozen_range_deferred`, `contig_frozen_range_taken`, `contig_frozen_range_failed` |
There are no recovery, “nothing” gate, or automatic-shrink counters in the
final series.
### `/proc/zoneinfo` and debugfs
- `/proc/zoneinfo` prints `free_nonmovable` for each zone.
- `/sys/kernel/debug/extfrag/gigablocks` prints one line per tracked gigablock:
its type count, last-observed content count and valid pageblock count. A
hotplug-disabled zone prints one disabled line.
- `extfrag/gigablock_contig` reports one gigablock per seq record: absent,
reserved, gigantic-hugetlb and in-use pageblocks, followed by the in-use page
classes when present. It holds the memory-hotplug read lock across the seq
operation and does not rescan a record merely because userspace used a small
buffer.
- Writing a block count to `extfrag/gigablock_evacuate` requests a debugfs pass
that bypasses the gate but still obeys the background attempt, check and visit
limits. Only debugfs passes print detailed evacuation lines to dmesg.
The debugfs files require `CONFIG_DEBUG_FS` and `CONFIG_COMPACTION`;
`gigablock_contig` also requires `CONFIG_CONTIG_ALLOC`.
---
## 10. Patch order
| # | Subject |
|---:|---|
| 1 | mm/page_alloc: count a merged buddy's pages under the merged type |
| 2 | mm/page_alloc: count guard pages as free again when their buddy merges |
| 3 | mm/page_alloc: count non-movable pageblocks per gigablock |
| 4 | mm/page_alloc: mark non-movable pageblocks that hold movable pages |
| 5 | mm/page_alloc: check a pageblock's pages before converting it to movable |
| 6 | mm: count movable pages added to non-movable pageblocks |
| 7 | mm/page_alloc: mark the pageblock where a movable page lands |
| 8 | mm/page_alloc: count crossings between the non-movable and movable sides |
| 9 | mm/page_alloc: convert any movable pageblock on a non-movable allocation |
| 10 | mm/page_alloc: refuse compaction capture of tracked movable blocks |
| 11 | mm/page_alloc: skip movable blocks during non-movable steals |
| 12 | mm/page_alloc: prefer claiming pageblocks from mixed gigablocks |
| 13 | mm/page_alloc: count full and partial gigablock scans |
| 14 | mm/page_alloc: count why a pageblock claim happens |
| 15 | mm/page_alloc: split the per-zone range scan out of the contiguous search |
| 16 | mm/compaction: centralize the alloc_contig isolation exception |
| 17 | mm/compaction: let a caller isolate the folio that fills a pageblock |
| 18 | mm/page_alloc: evacuate movable pages from mixed gigablocks |
| 19 | mm/page_alloc: disable the LRU caches only when a block is about to move |
| 20 | mm/page_alloc: skip a block that still holds what evacuation could not move |
| 21 | mm/compaction: evacuate mixed gigablocks when a claim finds no block |
| 22 | mm/page_alloc: ask for evacuation when a claim leaves movable pages behind |
| 23 | mm/page_alloc: concentrate non-movable claims in one gigablock |
| 24 | mm/page_alloc: count free pages on non-movable free lists |
| 25 | mm/page_alloc: seed a newly mixed gigablock with free blocks |
| 26 | mm/page_alloc: refill the target gigablock with whole blocks |
| 27 | mm/page_alloc: wake kcompactd only when a pass can help |
| 28 | mm/page_alloc: expire stale resistance records |
| 29 | mm/page_alloc: return excess free blocks to movable lists |
| 30 | mm/compaction: let kcompactd start an evacuation pass on its own |
| 31 | mm/page_alloc: skip unmixed gigablocks for an allocation with a fallback |
| 32 | mm/page_alloc: try the other non-movable type before unmixed gigablocks |
| 33 | mm/page_alloc: track pageblocks with non-movable pages seen by scans |
| 34 | mm/vmstat: report non-movable pageblocks per gigablock |
| 35 | mm/page_alloc: report what stops each gigablock supplying a gigantic page |
| 36 | mm/page_alloc: count the outcome of each range a contiguous alloc tries |
In groups: patch 1 repairs allocator accounting needed by the later free-list
limits; patches 2-10 establish gigablock accounting and close hidden crossings;
11-13 implement and measure bounded mixed-gigablock claims; 14-16 prepare the
contiguous and compaction paths; 17-21 add evacuation and its request paths;
22-25 add target placement and bounded supply; 26-32 add gate liveness,
resistance expiry, bounded release, autonomous kcompactd work, fallback
and last-resort ordering; 33-36 add scan-observed counts and
contiguous-range reporting and outcome counters.
