Keeping gigabytes whole: the gigablock design

The gigablock series is a 36-patch change to the Linux page allocator, based on v7.3-rc1. It concentrates non-movable pageblocks in a small number of 1 GiB-aligned ranges so that the remaining ranges can still supply gigantic hugetlb pages after the machine has been running for a long time.

Patch numbers refer to the ordered series in section 10. Measurements from older prototypes are labelled as such; they explain design choices but are not presented as measurements of this exact revision.

The code is in the riel/gigablock-2026-10-03 branch

Contents

1. The problem
2. Scope, terms and activation
3. State and accounting
4. Allocation policy and supply
5. Evacuation
6. Bounds, liveness and edge cases
7. Measurements and validation
8. Limits and deferred work
9. Observability
10. Patch order

1. The problem

A 1 GiB gigantic page needs a 1 GiB-aligned physical range in which every page is free or movable. One non-movable page can make the whole range unusable, even if almost all of it is free.

The buddy allocator organizes free memory by order and migratetype. Free pages in MIGRATE_UNMOVABLE and MIGRATE_RECLAIMABLE pageblocks normally serve kernel allocations; free pages in MIGRATE_MOVABLE pageblocks serve movable allocations. When the requested non-movable list has no sufficiently large buddy, the fallback path reaches the movable lists and chooses a pageblock by free-block size, without considering which 1 GiB range contains it.

Repeated fallbacks therefore spread a small amount of non-movable memory over many gigantic-page-sized ranges. The problem is placement, not necessarily the amount of non-movable memory. On a 765 GiB node after 53 days of build and filesystem work, the median 1 GiB range was 13.0% non-movable while the machine as a whole was 13.1% non-movable, and none of 754 ranges was clear of non-movable pageblocks. On a 253 GiB v7.3-rc1 test host, a metadata and memory workload mixed 251 of 254 ranges in 9.2 hours and a request for eight 1 GiB pages returned none.

The series changes the fallback decision. If a non-movable allocation must cross into movable memory, it first looks for a usable movable pageblock in a range that is already mixed. It also keeps a bounded reserve of free non-movable pageblocks in those ranges, migrates movable content out of non-movable pageblocks, and makes an unmixed range the last resort for callers that can wait or retry smaller.

flowchart TD
    A["non-movable allocation"] --> B{"own migratetype has a buddy<br/>of the requested order or larger?"}
    B -->|yes| C["allocate inside an existing<br/>non-movable pageblock"]
    B -->|no, stock behavior| D["claim the largest movable buddy<br/>without regard to 1 GiB location"]
    D --> E["mix another gigablock"]
    E --> F["repeat across the zone"]
    F --> G["non-movable content becomes<br/>thinly spread across most gigablocks"]
    B -->|no, gigablock policy| H["search already mixed gigablocks first"]
    H -->|usable block| I["claim without losing another<br/>gigantic-page candidate"]
    H -->|none| J["try other non-movable type,<br/>bounded evacuation, or caller fallback"]
    J -->|last resort| K["mix an unmixed gigablock<br/>rather than fail the allocation"]

A prerequisite fix at the bottom of the series restores free-page accounting for CONFIG_DEBUG_PAGEALLOC guard pages. When a guard buddy merges, clear_page_guard() no longer updates free counters; the merged block otherwise reaches a free list with NR_FREE_PAGES and nr_free_nonmovable short by the guard pages. The fix credits the cleared guard pages under the merged block's migratetype.

2. Scope, terms and activation

Terms

A gigablock is fixed at 1 GiB rather than PUD_SIZE. On arm64 with 16 KiB pages a PUD is 64 GiB, while 1 GiB is a run of 32 PMDs that the TLB can coalesce. Larger contiguous allocations require several aligned gigablocks.

Build and boot gate

The feature is capable only when all three configuration options are enabled:

CONFIG_CONTIG_ALLOC && CONFIG_HUGETLB_PAGE && CONFIG_ARCH_HAS_GIGANTIC_PAGE

gigablocks=off disables it at boot. A zone is tracked only when its boot-time, 1 GiB-aligned span covers at least two gigablocks and the per-gigablock metadata allocation succeeds. (A failed resistance-table allocation only warns; the zone stays tracked.)

Kexec HandOver boots stay tracked. Preserved pages get initialized struct pages and sit in memblock ranges the seed reads, so scans see them as ordinary reserved memory holding non-movable content.

Stable geometry and scan readiness

Metadata is allocated from the boot-time zone span. gigablock_zone_start_pfn and gigablock_zone_end_pfn remain fixed even if hotplug later changes the zone. Only pageblocks wholly inside that span, with both ends in the zone, are tracked; a hole inside the pageblock does not exclude it. A partial pageblock at either zone boundary is left outside tracking: it is never counted, and the steal refusal and no-mix/last-resort logic do not cover it. On arm64 with 64 KiB pages the untracked span can reach 512 MiB at each boundary.

The early boot seed estimates presence and migratetype before all struct pages exist. page_alloc_init_late() recounts each gigablock after deferred struct page initialization and hugetlb bootmem initialization, then enables a static key. Placement, seeding, release and evacuation do no page scans before that point.

The fixed metadata does not try to follow memory hotplug. Before memory is moved into or out of a tracked zone, gigablock_hotplug_off() permanently sets ZONE_GIGABLOCK_OFF, clears the target and disables all placement, marker and evacuation operations for that zone until reboot. Existing arrays remain allocated so in-flight readers stay within their original bounds. Debugfs reports the zone as disabled. This applies even if a hotplug operation later rolls back.

3. State and accounting

classDiagram
    direction LR
    class zone {
        nr_free_nonmovable
        gigablocks[]
        nr_gigablocks
        gigablock_scan
        gigablock_release_cursor
        gigablock_target
        claims_since_refill
        nr_mixed_gigablocks
        gigablock_evac_credits
        gigablock_borrow_epoch
        gigablock_swept_epoch
        gigablock_epoch_probe_after
        gigablock_epoch_probe_pending
        gigablock_evac_cursor
        gigablock_cycle
        gigablock_cycle_epoch
        gigablock_cycle_cleared
        gigablock_evac_pass
        gigablock_evac_block
        gigablock_evac_lock
    }
    class gigablock {
        nr_nonmovable_blocks
        nr_nonmovable_blocks_seen
        total_blocks
        scan_block
        scan_missed
        resisted[]
    }
    class pageblock_bits {
        migratetype
        PB_migrate_mixed
        PB_has_nonmovable
    }
    class task_struct {
        in_gigablock_putback
    }
    class pglist_data {
        kcompactd_evacuate_zones
    }
    zone "1" --> "many" gigablock : boot-time 1 GiB spans
    gigablock "1" --> "many" pageblock_bits : valid pageblocks
    pglist_data "1" --> "many" zone : kcompactd bitmap

Per-pageblock state

The series adds two pageblock bits:

PB_migrate_mixed       /* non-movable block may contain movable pages */
PB_has_nonmovable      /* last content scan found non-movable pages */

PB_migrate_mixed is a conservative hint. It is set when a claim leaves movable pages behind, when a movable allocation takes from a non-movable free list, when highatomic conversion can strand movable content, when __isolate_free_page() keeps a block non-movable, when compaction capture puts a movable request in a non-movable block, and when an LRU arrival is observed in a non-movable block. An evacuation scan repairs a missing hint. The bit may remain set after the movable pages leave; that costs a scan but does not hide work.

The bit is cleared when the pageblock becomes movable, when a whole-block buddy is added to a free list, or when compaction captures a whole block, which bypasses the free list. Moving a pageblock to a movable migratetype and clearing the bit use one masked pageblock-flags update, so a racing arrival cannot leave the type and marker in an impossible intermediate state. The mixed-marker mutators are no-ops for an untracked or hotplug-disabled zone.

There is deliberately no aggregate nr_borrowed_blocks. Maintaining an exact level across lockless LRU arrivals and type changes introduced a race between the bit and the aggregate. Liveness is carried by the borrow and swept epochs; the pass always scans both marked and unmarked non-movable pageblocks.

PB_has_nonmovable is a last-observed content snapshot. Evacuation sets or clears it after scanning a non-movable pageblock, and whole-block free clears it. It can miss an unscanned block and can retain a stale positive until the next scan. Placement does not use it; diagnostics print the per-gigablock aggregate derived from it.

Hugetlb folios are counted separately. They are movable for contiguous-range analysis, but gigablock evacuation has no hugetlb isolation path, so they are excluded from the actionable movable count. The post-migration recount still sees them, which prevents a block containing hugetlb pages from being reported as cleared.

Per-gigablock state

struct gigablock {
        unsigned int nr_nonmovable_blocks;
        unsigned int nr_nonmovable_blocks_seen;
        unsigned int total_blocks;
        unsigned int scan_block;
        unsigned int scan_missed;
        u32 *resisted;
};

nr_nonmovable_blocks is maintained at the pageblock type-change site under zone->lock. The first non-movable pageblock increments zone->nr_mixed_gigablocks; removal of the last decrements it.

total_blocks contains only pageblocks the common validity predicate can walk whole. scan_block resumes the next mixed-gigablock claim inside this range. scan_missed accumulates misses across bounded scans and records a full sweep only after the cursor has passed every pageblock since the last hit.

resisted[] is one u32 per pageblock when CONFIG_COMPACTION is enabled. The low 16 bits store the actionable movable-page count, the high 16 bits the evacuation cycle, zero means no record, and U32_MAX means armed but not recorded. A matching record suppresses attempts in its recording cycle and the next three, so a permanently unchanged block is retried once every four cycles. A new allocator mark, LRU arrival, type change or whole-block free clears it.

Per-zone and per-task state

The main zone fields are:

task_struct->in_gigablock_putback prevents an evacuator's own failed-folio putbacks from looking like new arrivals.

pgdat->kcompactd_evacuate_zones is the per-node bitmap of zones with requested or autonomous evacuation work. There is no per-node shrink state or work item.

4. Allocation policy and supply

4.1 Closing unaccounted crossings

stateDiagram-v2
    direction LR
    [*] --> Movable
    Movable --> NonmovableWhole : whole-run claim, seed, or refill
    Movable --> NonmovableBorrowed : partly used claim; set mixed hint
    NonmovableWhole --> NonmovableBorrowed : movable allocation or LRU arrival
    NonmovableBorrowed --> NonmovableWhole : evacuation clears movable content; hint may remain
    NonmovableWhole --> Movable : release or checked movable claim
    NonmovableBorrowed --> NonmovableBorrowed : movable conversion refused while kernel content remains
    Movable --> Movable : non-movable steal or capture refused

The accounting is useful only if every non-movable use of a tracked movable pageblock reaches a type change or is refused. The series closes three paths:

1. __rmqueue_steal() does not let a non-movable allocation take individual pages from a tracked movable pageblock while leaving its type unchanged.
2. A non-movable claim converts every pageblock in the free run it takes. A partly used pageblock is converted and marked PB_migrate_mixed.
3. compaction_capture() refuses a tracked movable block to a non-movable request. The block reaches the movable free list and the allocation retry goes through the normal claim path, which either converts and counts it or refuses it. Other captures update the mixed marker consistently.

The explicit movable-conversion paths inspect content first. __isolate_free_page() keeps a failing block non-movable, marks it mixed and counts gigablock_keep_nonmovable. Highatomic unreserve keeps such a block reserved on its first pass; later passes, including the forced pass before OOM, release it as MIGRATE_UNMOVABLE.

4.2 Mixed-gigablock claim search

flowchart TD
    A["non-movable allocation"] --> O{"own free list serves<br/>requested order?"}
    O -->|yes| DONE["allocate"]
    O -->|no| WALK["walk orders from MAX_PAGE_ORDER down;<br/>other non-movable type precedes movable"]
    WALK --> F{"fallback selected<br/>at this order?"}
    F -->|other non-movable type| CLAIM["claim if compatible"]
    CLAIM -->|served| DONE
    CLAIM -->|not claimable| MORE
    F -->|first movable fallback| Q{"request below pageblock_order<br/>and scans ready?"}
    Q -->|yes| M["try target, then round-robin<br/>eligible mixed gigablocks<br/>4 content / 256 block / 64 region limits"]
    M -->|buddy of requested order or larger| MC["claim whole run or one used block;<br/>mark movable content"]
    MC --> DONE
    M -->|none| EV["request evacuation if gate opens"]
    EV --> C
    Q -->|no| C{"movable candidate is in<br/>an unmixed tracked gigablock?"}
    C -->|no| CLAIM
    C -->|HAS_FALLBACK| REF["refuse candidate"]
    C -->|otherwise, first pass| SAVE["remember skipped candidate"]
    REF --> MORE{"more orders?"}
    SAVE --> MORE
    F -->|none| MORE
    MORE -->|yes| WALK
    MORE -->|no| NF{"ALLOC_NOFRAGMENT?"}
    NF -->|no| STEAL["try steal fallbacks;<br/>tracked movable blocks are refused,<br/>untracked movable blocks remain eligible"]
    NF -->|yes| AFTER
    STEAL -->|served| DONE
    STEAL -->|none| AFTER{"ordinary skipped<br/>candidate remembered?"}
    AFTER -->|yes| LAST["last-resort claim without<br/>repeating mixed search"]
    LAST -->|served| SEED["claim unmixed block; seed if headroom permits"]
    LAST -->|not claimable| RET
    SEED --> DONE
    AFTER -->|no| RET["return NULL;<br/>caller falls back or tries next zone"]

For requests below pageblock_order, the first movable fallback causes one mixed-gigablock search:

1. Try the target first, with half of the four content scans and half of the 256 pageblock checks reserved for the rest of the zone.
2. Continue round-robin from zone->gigablock_scan through other eligible mixed gigablocks.
3. Accept a movable pageblock only if it contains a buddy of the requested order or larger. A sum of smaller buddies cannot serve the allocation on this path.
4. Take a buddy of pageblock_order or larger whole, retyping every pageblock in the run. Otherwise convert the one pageblock, move its free pages to the requested non-movable type, mark any movable content, and allocate from it.
5. If the search misses, continue the ordinary size-ordered fallback. The series never fails an allocation merely because placement could not help.

Each claim has three independent limits:

Unmixed, empty and fully non-movable descriptors do not consume the 64 eligible visits. Charging them would create a false 64 GiB placement horizon on a large zone and could mix a clean gigablock while a usable mixed one existed later. The descriptor walk can therefore cover the zone; the separate pageblock and content limits bound the expensive work. An exact-tip function-graph run under claim load recorded a 6.329-microsecond maximum in gigablock_alloc_flags(). It did not attribute the mixed search itself, and that build had no IRQ-off tracer, so very large-zone IRQ-off cost remains a measurement concern.

The target is chosen by a full scan only during boot and the late recount. At runtime no target replacement scans the zone: an empty target is invalidated by the type transition, a full target is invalidated when its next claim misses, and the next successful round-robin claim elects its gigablock. A type change that creates a partly mixed gigablock also elects it when no target exists. A fragmented, partly mixed target can still miss repeatedly, but it receives only half of the per-claim pageblock and content budgets.

4.3 Refusing an unmixed gigablock

One allocation class can postpone the last-resort crossing: ALLOC_HAS_FALLBACK, a higher-order __GFP_NORETRY allocation with a smaller caller-level fallback. It skips the unmixed block independent of the gate, because the smaller retry can use existing non-movable free memory.

At each order the other non-movable type is tried before movable. If all safer choices fail, an allocation without a fallback still takes the unmixed gigablock; preservation must not turn into allocation failure.

Bulk refills set no flag, so only the universal last-resort rule steers them; a refill that enters steal mode without a claim ends short.

4.4 Seed, refill and release

flowchart TD
    MIX["claim first mixes a gigablock"] --> HS{"watermark and one-third<br/>share leave headroom?"}
    HS -->|yes| SEED["seed wholly free movable runs"]
    HS -->|no| HOLD1["do not seed"]
    CLAIMS["64 qualifying smaller-only claims"] --> HR{"watermark and one-third<br/>share leave headroom?"}
    HR -->|yes, target exists| FILL["refill wholly free runs in target"]
    HR -->|yes, no target| HOLD2["wait for a target"]
    HR -->|no| EXCESS{"pool is above high watermark?"}
    EXCESS -->|yes| RELEASE["choose one of at most 16 round-robin candidates;<br/>release up to 32 blocks from it"]
    EXCESS -->|no| HOLD3["nothing to release"]
    EVAC["evacuation clears movable content<br/>from a non-movable block"] --> POOL["non-movable free lists<br/>nr_free_nonmovable"]
    SEED --> POOL
    FILL --> POOL
    POOL --> ALLOC["future non-movable allocations<br/>avoid another crossing"]
    RELEASE --> MOV["movable free lists"]

A claim that first mixes a gigablock may seed it with wholly free movable pageblocks. Every 64th qualifying smaller-only claim may refill the target. Both are limited by one snapshot of two headroom values:

• free non-movable pages stay below the zone's high watermark;
• free non-movable pages stay below one third of all free pages;
• no call intentionally takes more than 512 pageblocks, though the final buddy run is moved whole and can cross that count.

Whole-run conversion matters. Moving the free pages of a multi-pageblock buddy without changing every covered pageblock's type would put movable-typed blocks on a non-movable free list and break both accounting and fallback behavior.

Free pages later accumulate on the non-movable lists. When the pool is above the high watermark, nonmovable_release_excess() returns wholly free unmovable or reclaimable blocks to movable lists. It excludes the target, looks at at most 16 candidate gigablocks from a round-robin cursor, and returns at most 32 pageblocks per call, with the last whole buddy run allowed to cross the count. It records both a selected gigablock with no releasable block and a 16-entry window with no candidate. Release is called from the 64-claim refill interval even when no target exists, so a barren candidate cannot permanently block progress elsewhere.

This series does not invoke slab shrinkers. Shrinking the slab preemptively may help prevent spilling into more gigablocks, but the first version of the series should stay smaller.

5. Evacuation

In order to concentrate kernel allocations in a small number of gigablocks, it is necessary to free up pages in those gigablocks that were used by movable allocations. The current version of the series uses kcompactd to evacuate some page blocks inside gigablocks that have both movable and non-movable allocations.

5.1 Requests and the gate

flowchart TD
    A1["mixed search claims a partly used block"] --> G
    A2["mixed search finds no block"] --> G{"claim-time evacuation gate"}
    G -->|non-movable free at or above high| W["gate_wmark; no request"]
    G -->|non-movable share at or above one third| S["gate_share; no request"]
    G -->|swept epoch equals borrow epoch| E{"32-second claim probe due?"}
    E -->|no| EC["gate_epoch; no request"]
    E -->|yes| P["gate_probe; authorize one pass"]
    G -->|open| H{"zone still has a mixed gigablock?"}
    P --> H
    H -->|no| EMPTY["queue helper does nothing"]
    H -->|yes| QW["add saturating credit and zone bit"]
    QW --> WAKE{"rmqueue or bulk exit<br/>permits a wake?"}
    WAKE -->|yes, after zone lock drops| K["kcompactd admission"]
    WAKE -->|no| POLL["later wake or node poll"]
    POLL --> K
    K --> U{"watermark, share and epoch<br/>or pending probe allow work?"}
    U -->|no| KEEP["keep credits for a later pass"]
    U -->|yes| TAKE["take credits; request max of credits<br/>and high-watermark shortfall"]
    TAKE --> RUN["bounded evacuation pass"]
    RUN --> REFUND["refund credits not satisfied"]

Two events request evacuation, each only when the claim-time gate passes: a partly used claim made by the mixed-gigablock search, and a mixed-search miss. Size-ordered claims mark the block but do not request a pass. A request adds one credit with a saturating compare/exchange, up to 64, sets the zone bit in kcompactd_evacuate_zones, and defers the actual wake until after zone->lock is released. rmqueue() wakes only for allocations carrying ALLOC_KSWAPD, and bulk exit only for __GFP_KSWAPD_RECLAIM. ALLOC_HAS_FALLBACK requests therefore wait for a later wake-capable allocation or the kcompactd poll.

The gate evaluates these tests in order:

1. watermark: stop if nr_free_nonmovable already meets the zone high watermark;
2. share: stop if the non-movable lists hold at least one third of all free pages;
3. epoch: stop if a complete cycle cleared nothing and no observed movable arrival advanced the epoch since that cycle began.

There is no nr_borrowed_blocks or “nothing” gate. The mixed marker is a hint, and pass 1 deliberately scans unmarked non-movable pageblocks, so an aggregate of markers was neither necessary nor race-free.

The epoch gate has a bounded escape. A claim rejected only by the epoch test starts a 32-second deadline. A later claim after the deadline records gigablock_evac_gate_probe and authorizes one pass. The probe bypasses only the epoch test: watermark and share are evaluated first. kcompactd polling honors a pending probe but cannot create one by itself.

Wake-capable requests wake kcompactd immediately. Tracked nodes also poll at 500 milliseconds with proactive compaction enabled, backing off to 32 seconds when a proactive pass makes no progress; with proactive compaction disabled the poll is 32 seconds. A closed gate leaves the credits in place for a later poll or wake. For each zone whose gate is open, kcompactd serves the larger of queued credits and the pageblock shortfall below the high watermark, capped at 64. The gate is checked before credits are consumed. Credits a bounded pass did not satisfy are returned through the same saturating helper.

5.2 One pass

flowchart TD
    START["hold gigablock_evac_lock;<br/>resume shared gigablock, phase, and block cursors"] --> STOP{"one cycle or visit limit, requested clears,<br/>fatal signal, zone disabled, or -ENOMEM?"}
    STOP -->|yes| SAVE["save first unchecked block<br/>or next unvisited gigablock"]
    STOP -->|no| MIX{"gigablock is mixed<br/>and has valid pageblocks?"}
    MIX -->|no| NEXT["advance gigablock cursor"]
    MIX -->|yes| PHASE{"saved phase"}
    PHASE -->|0| MARKED["scan marked non-movable blocks"]
    PHASE -->|1| UNMARKED["scan unmarked non-movable blocks"]
    MARKED --> CAND["check one pageblock in one IRQ-off section"]
    UNMARKED --> CAND
    CAND --> LIMIT{"check or attempt limit reached?"}
    LIMIT -->|yes| SAVE
    LIMIT -->|no| COUNT["count kernel, reserved, and actionable<br/>movable pages net of hugetlb"]
    COUNT -->|unchanged resistance record| SKIP["skip until record expires"]
    COUNT -->|no actionable movable pages| STALE["count stale"]
    COUNT -->|actionable pages| MIG["repair hint; arm record;<br/>drain LRU batching; migrate"]
    MIG --> RECOUNT["recount with hugetlb included"]
    RECOUNT -->|no movable page remains| CLEAR["clear one requested block"]
    RECOUNT -->|movable pages remain| STUCK["record stuck reason;<br/>an error reports aborted"]
    STUCK --> FATAL{"migration error is -ENOMEM?"}
    FATAL -->|yes| SAVE
    FATAL -->|no| MORE
    SKIP --> MORE{"more blocks in this phase?"}
    STALE --> MORE
    CLEAR --> MORE
    MORE -->|yes| CAND
    MORE -->|no, phase 0| SWITCH["switch to unmarked phase"]
    SWITCH --> UNMARKED
    MORE -->|no, phase 1| NEXT
    NEXT --> STOP

The pass walks mixed gigablocks from a shared cursor. Within each gigablock it scans non-movable pageblocks in two phases: blocks marked PB_migrate_mixed, then unmarked blocks. It checks one pageblock in each IRQ-off section and reschedules between checks.

For a candidate block it:

1. counts non-movable, reserved and actionable movable pages, hugetlb netted out, under zone->lock and refreshes PB_has_nonmovable;
2. skips an unchanged resistance record;
3. repairs a missing mixed marker;
4. drains LRU batching lazily, only before the first migration attempt;
5. isolates and migrates movable folios without zone->lock;
6. puts failed folios back after each batch while suppressing false arrival accounting;
7. recounts the block and calls it cleared when no movable page remains, hugetlb included; otherwise it records the stuck reason.

Passes use MIGRATE_SYNC_LIGHT, disable LRU caches globally from the first migration attempt until the pass ends, and may record resistance.

The series does not empty and claim movable pageblocks as an evacuation fallback. That recovery experiment was dropped because it migrates from a block that remains movable while migration runs, so destination allocations can reuse the just-freed source pages. Its aggregate free-page precheck guaranteed neither a destination of the needed order nor a coalesced pageblock buddy.

5.3 Bounds and resume state

A kcompactd or debugfs pass tries migration in at most 64 blocks, checks at most 256 pageblocks and visits at most 64 gigablocks.

A failed or stale candidate consumes a check but not a requested clear. The attempt bound limits expensive migration even when nothing clears; the check and visit bounds keep LRU batching and the shared mutex from spanning a zone.

If a bound, signal or -ENOMEM stops the pass inside a gigablock, the saved phase and pageblock point to the first unchecked work. If the gigablock is finished, the gigablock cursor points to the next unvisited one. This avoids restarting at a resistant prefix.

5.4 Resistance and cycles

stateDiagram-v2
    [*] --> None
    None --> Armed : pass before migration
    Armed --> Recorded : cmpxchg stores actionable count and cycle
    Armed --> None : block cleared or concurrent arrival clears slot
    Armed --> Armed : migration error stores nothing
    Recorded --> None : new mark, LRU arrival, type change, or whole-block free
    Recorded --> Recorded : same count in recording cycle plus next three
    Recorded --> Armed : nonzero count changed or record expired

Before a migration the pass writes GIGABLOCK_RESIST_ARMED. If no arrival clears it while the lock is dropped, the pass stores the actionable movable-page count and cycle with cmpxchg(). A matching record suppresses another attempt in its recording cycle and the next three; the block is retried once every four cycles. Different content with the same count cannot hide behind the record because arrivals and type changes clear it first. A migration error leaves the armed value, which matches no count.

The unbounded cursor value identifies the cycle as cursor / nr_gigablocks. A cycle records its starting borrow epoch.

sequenceDiagram
    participant E as evacuation cursor
    participant Z as zone cycle state
    participant A as movable arrivals
    E->>Z: enter cycle, cycle_epoch = borrow_epoch, cycle_cleared = false
    A->>Z: new mixed mark increments borrow_epoch
    A->>Z: first arrival in a cycle advances epoch with cmpxchg
    E->>Z: clear a block, cycle_cleared = true
    E->>Z: cross into next cycle
    alt finished cycle cleared nothing
        E->>Z: swept_epoch = prior cycle_epoch
        alt borrow_epoch still equals swept_epoch
            Note over Z: epoch gate closes
        else an arrival advanced borrow_epoch
            Note over Z: epoch gate remains open
        end
    else finished cycle cleared work
        Note over Z: epoch gate remains open
    end

When a cycle finishes without clearing a block, it copies the cycle's start epoch to gigablock_swept_epoch. If no arrival advanced gigablock_borrow_epoch, the equal values close the epoch gate. A movable arrival advances the borrow epoch once per cycle, so an arrival the cursor already passed still keeps the gate open for another traversal.

6. Bounds, liveness and edge cases

Lock and latency bounds

• The mixed claim search reads at most four pageblocks and checks at most 256. Its 64-region budget charges only eligible mixed gigablocks; descriptor-only skips remain a large-zone measurement concern.
• Seed/refill and release hold zone->lock, but operate on one gigablock at a time and have explicit block budgets. Release candidate selection looks at 16 gigablocks, not the whole zone.
• Runtime target replacement never scans the zone.
• Evacuation checks one pageblock per IRQ-off section and has attempt, check and visit limits.
• extfrag/gigablock_contig uses a private seq iterator and emits one gigablock per record, so a small userspace read buffer does not rescan the same gigablock's pages.

Allocation safety

• Skipping an unmixed gigablock never fails an allocation by itself. A fallback caller retries at a smaller order; every other allocation takes the block on the last-resort claim.
• gigablocks=off, an incapable build and a hotplug-disabled zone make mixed marker mutators and all policy entry points no-ops.

Hotplug, holes and boundaries

The boot-time metadata is not resized. Hotplug disables the affected tracked zone before its span changes, and all type updates, markers, claims, refill, release, evacuation and diagnostics honor that state. The gigablock listing reports “tracking disabled after memory hotplug”; the contig reader skips the disabled zone.

The common validity rule requires both ends of a pageblock to be in the same zone and online before runtime scans touch it. Holes and partial boundary blocks remain outside the counts. This costs capacity at boundaries but avoids walking uninitialized or foreign struct pages.

Gigantic hugetlb pages

Evacuation's compaction control uses whole_block_folio rather than alloc_contig, so it can isolate an ordinary pageblock-sized folio without entering the hugetlb dissolve path. Hugetlb folios have no gigablock evacuation path. They are excluded from actionable movable counts and resistance fingerprints, but retained in the final recount. A block that also held actionable pages can be reported as stuck reason hugetlb; a block whose only movable count is hugetlb is stale and is not migrated. gigablock_contig reports huge only for a folio of at least 1 GiB. On architectures whose hstate supports migration, smaller hugetlb folios pass its unmovable-page test and are not blockers. alloc_contig_range() behavior is unchanged.

Disabled automatic slab shrink

An automatic global/root slab pass was implemented and tested, but not retained. On matched 256 GiB hosts creating 39.6 million empty files, the treatment ran 3,120 passes, freed 392 objects in 316,812 microseconds, and did not reduce slab, mixed-gigablock or non-movable-pageblock growth. The final series contains no shrink work item, memcg iteration, shrink state, vmstat events or tracepoint. Slab remains ordinary reclaim's responsibility.

7. Measurements and validation

Historical placement soak

Two identical single-node 256 GiB Bergamo machines ran an earlier prototype for 22.1 hours under anonymous memory, tmpfs, 64 GiB of page cache and a slow stream of dentries, sockets and page tables. That prototype differed materially from the final series: claim admission used total free pages rather than a buddy of the requested order, frozen kernel pages were misclassified, failed migration folios were retried within a block, the evacuator's own putbacks looked like arrivals, and pass bounds and target handling were older.

Stock v7.3-rc1 on the same class of hardware and workload mixed 251 of 254 gigablocks in 9.2 hours and supplied none of eight requested 1 GiB pages. A matched 17-hour run mixed 252 of 254 and supplied 15 of 100 requests. These results establish the placement effect, not the exact cost or behavior of the final implementation.

Bulk wake test

A 250 GiB test host running a recent version of the code made 4,096 calls to alloc_pages_bulk(GFP_NOWAIT | __GFP_NOWARN) for 512 pages each. GFP_NOWAIT carries __GFP_KSWAPD_RECLAIM, which the later wake condition requires. Of those, 312 calls queued evacuation work. Every one observed a wake before the bulk call returned; first request to first wake was at most 143 microseconds and 36.9 microseconds on average. Afterwards the host allocated and released eight 1 GiB pages without a warning. This predates the final fold but exercises the bulk wake mechanism retained in it.

Final revision validation

The final validation covered:

• every exact final tree built per commit for CONFIG_COMPACTION=y, CONFIG_COMPACTION=n, and CONFIG_COMPACTION=n with CMA enabled;
• final bzImage builds for those configurations and CONFIG_MEMCG=n;
• checkpatch with no errors (two quoted-string-split warnings in one zoneinfo hunk); changelog, static and function-growth checks;
• exact-tip warning-free VM ramfs, churn, wake, contiguous-allocation, baseline, hugetlb and duration-control workloads; the exact-tip config had lockdep and DEBUG_VM disabled;
• a separate CONFIG_ACPI_HOTPLUG_MEMORY=y kernel with memhp_default_state=offline: 2 GiB add, explicit online into tracked Normal, permanent tracking disable, offline, device removal, seq output, evacuation and contig reads. The default auto-online run placed the DIMM in untracked ZONE_MOVABLE and correctly did not exercise the disable path. QMP removed the DIMM device in both runs, while immediate object-del of its backend returned “in use”;
• exact-tip bare-metal boot, metadata walk, evacuation request and four 1 GiB hugepage allocations with no kernel warning. The evacuation request scanned 155 pageblocks and cleared none.
• an exact-tip KHO round-trip in a 16 GiB VM (test_kho, 1 GiB preserved, PROVE_LOCKING + DEBUG_VM): first boot tracked, kexec into a KHO boot still tracked, KHO restore succeeded, a debugfs evacuation write ran clean, and the serial log had no warning, lockdep report or oops.

The ENOMEM harness completed, but the build lacked fail_page_alloc; its four warning matches were intentional SysRq-L traces. Exact-tip function-graph maxima were 45.298 microseconds for background evacuation and 6.329 microseconds during claim load in gigablock_alloc_flags(). Wall-clock time was 0.56 milliseconds for the evacuation write. The build had no IRQ-off tracer, so these are function and wall-time measurements rather than IRQ-off bounds. The VM, hotplug and bare-metal runs above used the 39-patch pre-drop tip c3b15041d6c5; no direct pass ran in any of them, and the per-commit and bzImage builds were re-run on this exact tip.

8. Limits and deferred work

Accepted limits

These are behaviors the series ships with, documented so reviewers know what not to expect.

• The current test evidence does not include injected fail_page_alloc ENOMEM.
• Skipping an unmixed gigablock can move the allocation to the next zone in the zonelist, a lower local zone or a remote node, before the caller falls back or the last-resort claim runs. That ordering has no locality preference.
• Movable-typed pageblocks are never evacuation candidates. A non-movable allocation that finds a free page in one converts the whole block, marks it mixed, and evacuation then moves the movable strays out — but a block that is 100% full of page cache never enters that chain until reclaim frees something in it. Migrating a movable block's contents out without needing a free page first (the shelved recovery experiment) would close that gap.
• Orders at or above pageblock_order use the size-ordered claim path rather than the mixed-gigablock search.
• Highatomic reservation and cross-type buddy adoption can still choose a gigablock without placement policy. Their type changes remain accounted.
• A movable page allocated with __GFP_MOVABLE that never reaches the LRU or registers movable_ops can remain non-movable content in a movable block, with nothing identifying the producer.
• Resistance is a count fingerprint, not a classification of every migration failure. A permanently stuck block is retried once every four cycles.
• Release is driven by the 64-claim refill interval. A zone with no qualifying claims does not run release autonomously.
• Dynamic hotplug metadata is not implemented. A span change disables the affected zone until reboot.
• Partial boundary pageblocks are outside tracking, counting and placement protection; the untracked span can reach 512 MiB at each boundary on arm64 with 64 KiB pages.

Placement and search follow-ups

• Untracked edge and hole-adjacent blocks serve kernel demand only through the normal fallback order. Trying them first, since they can never form part of a gigantic page, remains future work.
• The mixed-gigablock claim search scans in the allocation hot path under zone->lock with interrupts off. Moving that work into watermark-maintenance slow paths, so claims need no hot-path scan, remains future work. Reducing claim-ahead overhead in the fast path is part of the same effort.
• A fragmented, partly mixed target can consume its half of the search budget on repeated misses. Runtime replacement is bounded and event-driven, not an order-specific target cache; whether an order-aware cache pays for itself is unmeasured.
• The eligible-only 64-gigablock budget does not bound descriptor reads across a very large zone. The expensive content and pageblock work is bounded; the descriptor-only worst case remains a measurement gate.
• nonmovable_release_excess() picks a gigablock by typed-block count, which says nothing about whether it holds wholly free blocks, then scans it. Walking the non-movable free lists at pageblock_order and above instead, returning whole free runs until back under the threshold with no gigablock selection, remains future work.
• Block device cache is allocated as UNMOVABLE, but has a .migrate_folio callback. If a system does lots of raw block device IO, that can result in many gigablocks becoming unusable for gigantic pages, for no good reason.

Evacuation follow-ups

• The evacuation pass and its helpers live in mm/page_alloc.c. Cleaning them up and moving them to a better home remains future work.
• Evacuation requests name a zone, and each pass scans from the zone cursor. Requesting evacuation for a particular gigablock, so passes go straight to the work, remains future work. Once evacuation and compaction are more targeted, the compaction epoch can be replaced with something smarter.
• Background evacuation reuses kcompactd, and the request, wake, gate and credit plumbing around it is larger than the earlier standalone evacuator. Two simplifications are on the table: separate evacuation code instead of reusing kcompactd, or async evacuation from a work item on the dropped direct-evacuation shape, either of which would shed most of the plumbing. Direct evacuation itself was dropped before the first posting for lack of test coverage.

Tech debt

• page_alloc.c is already too large, things should probably be split out somehow.

Series-order and naming follow-ups

• Move "mm/page_alloc: track pageblocks with non-movable pages seen by scans" and "mm/vmstat: report non-movable pageblocks per gigablock" earlier in the series, so reviewers see before-and-after results with the other allocator changes.
• Go over all the code and comments again, to make sure terminology is consistent with the rest of mm/
• Extract the fixes that can go upstream indendently of this series.
• Move gigablock slow path code into its own gigablock.c file.

9. Observability

/proc/vmstat

There are no recovery, “nothing” gate, or automatic-shrink counters in the final series.

/proc/zoneinfo and debugfs

• /proc/zoneinfo prints free_nonmovable for each zone.
• /sys/kernel/debug/extfrag/gigablocks prints one line per tracked gigablock: its type count, last-observed content count and valid pageblock count. A hotplug-disabled zone prints one disabled line.
• extfrag/gigablock_contig reports one gigablock per seq record: absent, reserved, gigantic-hugetlb and in-use pageblocks, followed by the in-use page classes when present. It holds the memory-hotplug read lock across the seq operation and does not rescan a record merely because userspace used a small buffer.
• Writing a block count to extfrag/gigablock_evacuate requests a debugfs pass that bypasses the gate but still obeys the background attempt, check and visit limits. Only debugfs passes print detailed evacuation lines to dmesg.

The debugfs files require CONFIG_DEBUG_FS and CONFIG_COMPACTION; gigablock_contig also requires CONFIG_CONTIG_ALLOC.

10. Patch order

In groups: patch 1 repairs allocator accounting needed by the later free-list limits; patches 2-10 establish gigablock accounting and close hidden crossings; 11-13 implement and measure bounded mixed-gigablock claims; 14-16 prepare the contiguous and compaction paths; 17-21 add evacuation and its request paths; 22-25 add target placement and bounded supply; 26-32 add gate liveness, resistance expiry, bounded release, autonomous kcompactd work, fallback and last-resort ordering; 33-36 add scan-observed counts and contiguous-range reporting and outcome counters.