From: "Barry Song (Xiaomi)" <baohua@kernel.org>
To: akpm@linux-foundation.org, linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org,
"Barry Song (Xiaomi)" <baohua@kernel.org>,
Lance Yang <lance.yang@linux.dev>,
Xueyuan Chen <xueyuan.chen21@gmail.com>,
Pedro Falcato <pfalcato@suse.de>,
Kairui Song <kasong@tencent.com>, Qi Zheng <qi.zheng@linux.dev>,
Shakeel Butt <shakeel.butt@linux.dev>,
wangzicheng <wangzicheng@honor.com>,
Suren Baghdasaryan <surenb@google.com>,
Lei Liu <liulei.rjpt@vivo.com>,
Matthew Wilcox <willy@infradead.org>,
Axel Rasmussen <axelrasmussen@google.com>,
Yuanchu Xie <yuanchu@google.com>, Wei Xu <weixugc@google.com>,
Will Deacon <will@kernel.org>,
Kalesh Singh <kaleshsingh@google.com>
Subject: [PATCH v3] mm/mglru: use folio_mark_accessed to replace folio_set_active
Date: Tue, 26 May 2026 21:09:38 +0800 [thread overview]
Message-ID: <20260526130938.66253-1-baohua@kernel.org> (raw)
MGLRU gives high priority to folios mapped in page tables. As a result,
folio_set_active() is invoked for all folios read during page faults. In
practice, however, readahead can bring in many folios that are never
accessed via page tables.
A previous attempt by Lei Liu proposed introducing a separate LRU for
readahead[1] to make readahead pages easier to reclaim, but that approach
is likely over-engineered.
Before commit 4d5d14a01e2c ("mm/mglru: rework workingset protection"),
folios with PG_active were always placed in the youngest generation,
leading to over-protection and increased refaults. After that commit,
PG_active folios are placed in the second youngest generation, which is
still too optimistic given the presence of readahead. In contrast, the
classic active/inactive scheme is more conservative.
This patch switches to using folio_mark_accessed() and
begins prefaulted file folios from the second oldest
generation instead of active generations.
We should also adjust the following accordingly:
- WORKINGSET_ACTIVATE: aligned with setting active for refaulted workingset
folios;
- lru_gen_folio_seq(): place (pre)faulted file folios into the second
oldest generation;
- promote second-scanned folios to workingset in
folio_check_references(): we now have to depend on
folio_lru_refs() > 1, since we previously relied on PG_referenced
being set during the first scan, but PG_referenced is now set
earlier.
On x86, running a kernel build inside a memcg with a 1GB memory
limit using 20 threads.
w/o patch:
real 1m50.764s
user 25m32.305s
sys 4m0.012s
pswpin: 1333245
pswpout: 4366443
pgpgin: 6962592
pgpgout: 17780712
swpout_zero: 1019603
swpin_zero: 14764
refault_file: 287794
refault_anon: 1347963
w/ patch:
real 1m48.879s
user 25m29.224s
sys 3m37.421s
pswpin: 568480
pswpout: 2322657
pgpgin: 4073416
pgpgout: 9613408
swpout_zero: 593275
swpin_zero: 9118
refault_file: 262505
refault_anon: 577550
active/inactive LRU:
real 1m49.928s
user 25m28.196s
sys 3m40.740s
pswpin: 463452
pswpout: 2309119
pgpgin: 4438856
pgpgout: 9568628
swpout_zero: 743704
swpin_zero: 7244
refault_file: 562555
refault_anon: 470694
Lance and Xueyuan made a huge contribution to this patch through testing.
[1] https://lore.kernel.org/linux-mm/20250916072226.220426-1-liulei.rjpt@vivo.com/
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
Tested-by: Lance Yang <lance.yang@linux.dev>
Tested-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Kairui Song <kasong@tencent.com>
Cc: Qi Zheng <qi.zheng@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: wangzicheng <wangzicheng@honor.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Lei Liu <liulei.rjpt@vivo.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: Kalesh Singh <kaleshsingh@google.com>
---
-v3:
* Fix 2nd pte-scanned promotion in folio_check_references(), per Kairui;
* Restore anon folios behaviour in lru_gen_folio_seq(), per Kairui;
-v2:
https://lore.kernel.org/linux-mm/20260525123205.51874-1-baohua@kernel.org/
* Fix WORKINGSET_ACTIVATE - workingset will be set to active during refault;
* Avoid unconditional protecting anon folios in lru_gen_folio_seq();
* Also adjusted workingset set accordingly in folio_check_references().
-v1:
https://lore.kernel.org/linux-mm/20260418120233.7162-1-baohua@kernel.org/
-rfc was:
[PATCH RFC] mm/mglru: lazily activate folios while folios are really mapped
https://lore.kernel.org/linux-mm/20260225212642.15219-1-21cnbao@gmail.com/
include/linux/mm_inline.h | 2 +-
mm/swap.c | 16 +++++++++++++---
mm/vmscan.c | 6 +++++-
mm/workingset.c | 10 ++++++----
4 files changed, 25 insertions(+), 9 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index a171070e15f0..a8430a7ae054 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -247,7 +247,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
(folio_test_dirty(folio) || folio_test_writeback(folio))))
gen = MIN_NR_GENS;
else
- gen = MAX_NR_GENS - folio_test_workingset(folio);
+ gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
}
diff --git a/mm/swap.c b/mm/swap.c
index 5cc44f0de987..a44829dcde7a 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -509,10 +509,20 @@ void folio_add_lru(struct folio *folio)
folio_test_unevictable(folio), folio);
VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
- /* see the comment in lru_gen_folio_seq() */
+ /*
+ * For refaulted workingset folios, set PG_active so they
+ * can be added to active generations.
+ * For prefaulted file folios, folio_mark_accessed() sets
+ * PG_referenced so lru_gen_folio_seq() places them into
+ * the second oldest generation.
+ */
if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
- lru_gen_in_fault() && !(current->flags & PF_MEMALLOC))
- folio_set_active(folio);
+ lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
+ if (folio_test_workingset(folio))
+ folio_set_active(folio);
+ else if (!folio_test_referenced(folio))
+ folio_mark_accessed(folio);
+ }
folio_batch_add_and_move(folio, lru_add);
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index e452cb043d46..745a55a3f7de 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -848,7 +848,11 @@ static bool lru_gen_set_refs(struct folio *folio)
return false;
}
- set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
+ /* Promote on second access */
+ if (folio_lru_refs(folio) > 1)
+ set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset));
+ else
+ folio_mark_accessed(folio);
return true;
}
#else
diff --git a/mm/workingset.c b/mm/workingset.c
index 07e6836d0502..f351798e723a 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -319,11 +319,13 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]);
- /* see folio_add_lru() where folio_set_active() will be called */
- if (lru_gen_in_fault())
- mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
-
if (workingset) {
+ /*
+ * see folio_add_lru(), where folio_set_active() is
+ * called for workingset folios
+ */
+ if (lru_gen_in_fault())
+ mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
folio_set_workingset(folio);
mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + type, delta);
} else
--
2.34.1
reply other threads:[~2026-05-26 13:09 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260526130938.66253-1-baohua@kernel.org \
--to=baohua@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=kaleshsingh@google.com \
--cc=kasong@tencent.com \
--cc=lance.yang@linux.dev \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=liulei.rjpt@vivo.com \
--cc=pfalcato@suse.de \
--cc=qi.zheng@linux.dev \
--cc=shakeel.butt@linux.dev \
--cc=surenb@google.com \
--cc=wangzicheng@honor.com \
--cc=weixugc@google.com \
--cc=will@kernel.org \
--cc=willy@infradead.org \
--cc=xueyuan.chen21@gmail.com \
--cc=yuanchu@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.