* [PATCH 0/8] Convert huge_zero_page to huge_zero_folio
@ 2024-03-26 20:28 Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 1/8] sparc: Use is_huge_zero_pmd() Matthew Wilcox (Oracle)
` (7 more replies)
0 siblings, 8 replies; 11+ messages in thread
From: Matthew Wilcox (Oracle) @ 2024-03-26 20:28 UTC (permalink / raw
To: Andrew Morton; +Cc: Matthew Wilcox (Oracle), linux-mm
Almost all the callers of is_huge_zero_page() already have a folio.
And they should -- is_huge_zero_page() will return false for tail pages,
even if they're tail pages of the huge zero page. That's confusing,
and one of the benefits of the folio conversion is to get rid of this
confusion.
Matthew Wilcox (Oracle) (8):
sparc: Use is_huge_zero_pmd()
mm: Add is_huge_zero_folio()
mm: Add pmd_folio()
mm: Convert migrate_vma_collect_pmd to use a folio
mm: Convert huge_zero_page to huge_zero_folio
mm: Convert do_huge_pmd_anonymous_page to huge_zero_folio
dax: Use huge_zero_folio
mm: Rename mm_put_huge_zero_page to mm_put_huge_zero_folio
arch/sparc/mm/tlb.c | 6 ++--
fs/dax.c | 14 ++++----
fs/proc/page.c | 2 +-
include/linux/huge_mm.h | 14 ++++----
include/linux/pgtable.h | 2 ++
include/trace/events/fs_dax.h | 16 ++++-----
kernel/fork.c | 2 +-
mm/huge_memory.c | 65 ++++++++++++++++++-----------------
mm/madvise.c | 2 +-
mm/mempolicy.c | 4 +--
mm/migrate_device.c | 16 ++++-----
mm/mlock.c | 2 +-
mm/swap.c | 2 +-
mm/swap_state.c | 2 +-
mm/userfaultfd.c | 4 +--
15 files changed, 78 insertions(+), 75 deletions(-)
--
2.43.0
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 1/8] sparc: Use is_huge_zero_pmd()
2024-03-26 20:28 [PATCH 0/8] Convert huge_zero_page to huge_zero_folio Matthew Wilcox (Oracle)
@ 2024-03-26 20:28 ` Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 2/8] mm: Add is_huge_zero_folio() Matthew Wilcox (Oracle)
` (6 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Matthew Wilcox (Oracle) @ 2024-03-26 20:28 UTC (permalink / raw
To: Andrew Morton; +Cc: Matthew Wilcox (Oracle), linux-mm
There's no need to convert to a page, much less a folio. We can tell from
the pmd whether it is a huge zero page or not. Saves 60 bytes of text.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
arch/sparc/mm/tlb.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/arch/sparc/mm/tlb.c b/arch/sparc/mm/tlb.c
index b44d79d778c7..19642f7ffb52 100644
--- a/arch/sparc/mm/tlb.c
+++ b/arch/sparc/mm/tlb.c
@@ -183,12 +183,12 @@ static void __set_pmd_acct(struct mm_struct *mm, unsigned long addr,
* hugetlb_pte_count.
*/
if (pmd_val(pmd) & _PAGE_PMD_HUGE) {
- if (is_huge_zero_page(pmd_page(pmd)))
+ if (is_huge_zero_pmd(pmd))
mm->context.hugetlb_pte_count++;
else
mm->context.thp_pte_count++;
} else {
- if (is_huge_zero_page(pmd_page(orig)))
+ if (is_huge_zero_pmd(orig))
mm->context.hugetlb_pte_count--;
else
mm->context.thp_pte_count--;
@@ -259,7 +259,7 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
* Sanity check pmd before doing the actual decrement.
*/
if ((pmd_val(entry) & _PAGE_PMD_HUGE) &&
- !is_huge_zero_page(pmd_page(entry)))
+ !is_huge_zero_pmd(entry))
(vma->vm_mm)->context.thp_pte_count--;
return old;
--
2.43.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 2/8] mm: Add is_huge_zero_folio()
2024-03-26 20:28 [PATCH 0/8] Convert huge_zero_page to huge_zero_folio Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 1/8] sparc: Use is_huge_zero_pmd() Matthew Wilcox (Oracle)
@ 2024-03-26 20:28 ` Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 3/8] mm: Add pmd_folio() Matthew Wilcox (Oracle)
` (5 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Matthew Wilcox (Oracle) @ 2024-03-26 20:28 UTC (permalink / raw
To: Andrew Morton; +Cc: Matthew Wilcox (Oracle), linux-mm
This is the folio equivalent of is_huge_zero_page(). It doesn't add any
efficiency, but it does prevent the caller from passing a tail page and
getting confused when the predicate returns false.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
fs/proc/page.c | 2 +-
include/linux/huge_mm.h | 10 ++++++++++
mm/huge_memory.c | 6 +++---
mm/mempolicy.c | 2 +-
mm/swap.c | 2 +-
mm/swap_state.c | 2 +-
mm/userfaultfd.c | 2 +-
7 files changed, 18 insertions(+), 8 deletions(-)
diff --git a/fs/proc/page.c b/fs/proc/page.c
index d6953f95e3b4..5bc82828c6aa 100644
--- a/fs/proc/page.c
+++ b/fs/proc/page.c
@@ -155,7 +155,7 @@ u64 stable_page_flags(const struct page *page)
else if (folio_test_large(folio)) {
if ((k & PG_lru) || is_anon)
u |= 1 << KPF_THP;
- else if (is_huge_zero_page(&folio->page)) {
+ else if (is_huge_zero_folio(folio)) {
u |= 1 << KPF_ZERO_PAGE;
u |= 1 << KPF_THP;
}
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 1540a1481daf..600c6008262b 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -356,6 +356,11 @@ static inline bool is_huge_zero_page(const struct page *page)
return READ_ONCE(huge_zero_page) == page;
}
+static inline bool is_huge_zero_folio(const struct folio *folio)
+{
+ return READ_ONCE(huge_zero_page) == &folio->page;
+}
+
static inline bool is_huge_zero_pmd(pmd_t pmd)
{
return pmd_present(pmd) && READ_ONCE(huge_zero_pfn) == pmd_pfn(pmd);
@@ -485,6 +490,11 @@ static inline bool is_huge_zero_page(const struct page *page)
return false;
}
+static inline bool is_huge_zero_folio(const struct folio *folio)
+{
+ return false;
+}
+
static inline bool is_huge_zero_pmd(pmd_t pmd)
{
return false;
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 4556144cff09..78f9132daa52 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -789,12 +789,12 @@ struct deferred_split *get_deferred_split_queue(struct folio *folio)
}
#endif
-static inline bool is_transparent_hugepage(struct folio *folio)
+static inline bool is_transparent_hugepage(const struct folio *folio)
{
if (!folio_test_large(folio))
return false;
- return is_huge_zero_page(&folio->page) ||
+ return is_huge_zero_folio(folio) ||
folio_test_large_rmappable(folio);
}
@@ -3085,7 +3085,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
}
- is_hzp = is_huge_zero_page(&folio->page);
+ is_hzp = is_huge_zero_folio(folio);
if (is_hzp) {
pr_warn_ratelimited("Called split_huge_page for huge zero page\n");
return -EBUSY;
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 3fae74b62c51..6e7069ecf713 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -510,7 +510,7 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
return;
}
folio = pfn_folio(pmd_pfn(*pmd));
- if (is_huge_zero_page(&folio->page)) {
+ if (is_huge_zero_folio(folio)) {
walk->action = ACTION_CONTINUE;
return;
}
diff --git a/mm/swap.c b/mm/swap.c
index 500a09a48dfd..f72364e92d5f 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -985,7 +985,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
struct folio *folio = folios->folios[i];
unsigned int nr_refs = refs ? refs[i] : 1;
- if (is_huge_zero_page(&folio->page))
+ if (is_huge_zero_folio(folio))
continue;
if (folio_is_zone_device(folio)) {
diff --git a/mm/swap_state.c b/mm/swap_state.c
index bfc7e8c58a6d..2deac23633cd 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -301,7 +301,7 @@ void free_page_and_swap_cache(struct page *page)
struct folio *folio = page_folio(page);
free_swap_cache(folio);
- if (!is_huge_zero_page(page))
+ if (!is_huge_zero_folio(folio))
folio_put(folio);
}
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 163511ab5896..4f1a392fe84f 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -1699,7 +1699,7 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
!pmd_none(dst_pmdval)) {
struct folio *folio = pfn_folio(pmd_pfn(*src_pmd));
- if (!folio || (!is_huge_zero_page(&folio->page) &&
+ if (!folio || (!is_huge_zero_folio(folio) &&
!PageAnonExclusive(&folio->page))) {
spin_unlock(ptl);
err = -EBUSY;
--
2.43.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 3/8] mm: Add pmd_folio()
2024-03-26 20:28 [PATCH 0/8] Convert huge_zero_page to huge_zero_folio Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 1/8] sparc: Use is_huge_zero_pmd() Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 2/8] mm: Add is_huge_zero_folio() Matthew Wilcox (Oracle)
@ 2024-03-26 20:28 ` Matthew Wilcox (Oracle)
2024-04-04 19:39 ` David Hildenbrand
2024-03-26 20:28 ` [PATCH 4/8] mm: Convert migrate_vma_collect_pmd to use a folio Matthew Wilcox (Oracle)
` (4 subsequent siblings)
7 siblings, 1 reply; 11+ messages in thread
From: Matthew Wilcox (Oracle) @ 2024-03-26 20:28 UTC (permalink / raw
To: Andrew Morton; +Cc: Matthew Wilcox (Oracle), linux-mm
Convert directly from a pmd to a folio without going through another
representation first. For now this is just a slightly shorter way to
write it, but it might end up being more efficient later.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
include/linux/pgtable.h | 2 ++
mm/huge_memory.c | 6 +++---
mm/madvise.c | 2 +-
mm/mempolicy.c | 2 +-
mm/mlock.c | 2 +-
mm/userfaultfd.c | 2 +-
6 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 600e17d03659..09c85c7bf9c2 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -50,6 +50,8 @@
#define pmd_pgtable(pmd) pmd_page(pmd)
#endif
+#define pmd_folio(pmd) page_folio(pmd_page(pmd))
+
/*
* A page table page can be thought of an array like this: pXd_t[PTRS_PER_PxD]
*
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 78f9132daa52..8ee09bfdfdb7 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1816,7 +1816,7 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
goto out;
}
- folio = pfn_folio(pmd_pfn(orig_pmd));
+ folio = pmd_folio(orig_pmd);
/*
* If other processes are mapping this folio, we couldn't discard
* the folio unless they all do MADV_FREE so let's skip the folio.
@@ -2086,7 +2086,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
if (pmd_protnone(*pmd))
goto unlock;
- folio = page_folio(pmd_page(*pmd));
+ folio = pmd_folio(*pmd);
toptier = node_is_toptier(folio_nid(folio));
/*
* Skip scanning top tier node if normal numa
@@ -2663,7 +2663,7 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
* It's safe to call pmd_page when folio is set because it's
* guaranteed that pmd is present.
*/
- if (folio && folio != page_folio(pmd_page(*pmd)))
+ if (folio && folio != pmd_folio(*pmd))
goto out;
__split_huge_pmd_locked(vma, pmd, range.start, freeze);
}
diff --git a/mm/madvise.c b/mm/madvise.c
index 7625830d6ae9..1f77a51baaac 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -363,7 +363,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
goto huge_unlock;
}
- folio = pfn_folio(pmd_pfn(orig_pmd));
+ folio = pmd_folio(orig_pmd);
/* Do not interfere with other mappings of this folio */
if (folio_likely_mapped_shared(folio))
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 6e7069ecf713..4f5d0923af8f 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -509,7 +509,7 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
qp->nr_failed++;
return;
}
- folio = pfn_folio(pmd_pfn(*pmd));
+ folio = pmd_folio(*pmd);
if (is_huge_zero_folio(folio)) {
walk->action = ACTION_CONTINUE;
return;
diff --git a/mm/mlock.c b/mm/mlock.c
index 1ed2f2ab37cd..30b51cdea89d 100644
--- a/mm/mlock.c
+++ b/mm/mlock.c
@@ -378,7 +378,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr,
goto out;
if (is_huge_zero_pmd(*pmd))
goto out;
- folio = page_folio(pmd_page(*pmd));
+ folio = pmd_folio(*pmd);
if (vma->vm_flags & VM_LOCKED)
mlock_folio(folio);
else
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 4f1a392fe84f..f6267afe65d1 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -1697,7 +1697,7 @@ ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
/* Check if we can move the pmd without splitting it. */
if (move_splits_huge_pmd(dst_addr, src_addr, src_start + len) ||
!pmd_none(dst_pmdval)) {
- struct folio *folio = pfn_folio(pmd_pfn(*src_pmd));
+ struct folio *folio = pmd_folio(*src_pmd);
if (!folio || (!is_huge_zero_folio(folio) &&
!PageAnonExclusive(&folio->page))) {
--
2.43.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 4/8] mm: Convert migrate_vma_collect_pmd to use a folio
2024-03-26 20:28 [PATCH 0/8] Convert huge_zero_page to huge_zero_folio Matthew Wilcox (Oracle)
` (2 preceding siblings ...)
2024-03-26 20:28 ` [PATCH 3/8] mm: Add pmd_folio() Matthew Wilcox (Oracle)
@ 2024-03-26 20:28 ` Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 5/8] mm: Convert huge_zero_page to huge_zero_folio Matthew Wilcox (Oracle)
` (3 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Matthew Wilcox (Oracle) @ 2024-03-26 20:28 UTC (permalink / raw
To: Andrew Morton; +Cc: Matthew Wilcox (Oracle), linux-mm
Convert the pmd directly to a folio and use it. Turns four calls
to compound_head() into one.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
mm/migrate_device.c | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/mm/migrate_device.c b/mm/migrate_device.c
index b6c27c76e1a0..d40b46ae9d65 100644
--- a/mm/migrate_device.c
+++ b/mm/migrate_device.c
@@ -71,7 +71,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
return migrate_vma_collect_hole(start, end, -1, walk);
if (pmd_trans_huge(*pmdp)) {
- struct page *page;
+ struct folio *folio;
ptl = pmd_lock(mm, pmdp);
if (unlikely(!pmd_trans_huge(*pmdp))) {
@@ -79,21 +79,21 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
goto again;
}
- page = pmd_page(*pmdp);
- if (is_huge_zero_page(page)) {
+ folio = pmd_folio(*pmdp);
+ if (is_huge_zero_folio(folio)) {
spin_unlock(ptl);
split_huge_pmd(vma, pmdp, addr);
} else {
int ret;
- get_page(page);
+ folio_get(folio);
spin_unlock(ptl);
- if (unlikely(!trylock_page(page)))
+ if (unlikely(!folio_trylock(folio)))
return migrate_vma_collect_skip(start, end,
walk);
- ret = split_huge_page(page);
- unlock_page(page);
- put_page(page);
+ ret = split_folio(folio);
+ folio_unlock(folio);
+ folio_put(folio);
if (ret)
return migrate_vma_collect_skip(start, end,
walk);
--
2.43.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 5/8] mm: Convert huge_zero_page to huge_zero_folio
2024-03-26 20:28 [PATCH 0/8] Convert huge_zero_page to huge_zero_folio Matthew Wilcox (Oracle)
` (3 preceding siblings ...)
2024-03-26 20:28 ` [PATCH 4/8] mm: Convert migrate_vma_collect_pmd to use a folio Matthew Wilcox (Oracle)
@ 2024-03-26 20:28 ` Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 6/8] mm: Convert do_huge_pmd_anonymous_page " Matthew Wilcox (Oracle)
` (2 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Matthew Wilcox (Oracle) @ 2024-03-26 20:28 UTC (permalink / raw
To: Andrew Morton; +Cc: Matthew Wilcox (Oracle), linux-mm
With all callers of is_huge_zero_page() converted, we can now switch
the huge_zero_page itself from being a compound page to a folio.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
include/linux/huge_mm.h | 21 ++++++++-------------
mm/huge_memory.c | 28 ++++++++++++++--------------
2 files changed, 22 insertions(+), 27 deletions(-)
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 600c6008262b..7ba59ba36354 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -348,17 +348,12 @@ struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr,
vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf);
-extern struct page *huge_zero_page;
+extern struct folio *huge_zero_folio;
extern unsigned long huge_zero_pfn;
-static inline bool is_huge_zero_page(const struct page *page)
-{
- return READ_ONCE(huge_zero_page) == page;
-}
-
static inline bool is_huge_zero_folio(const struct folio *folio)
{
- return READ_ONCE(huge_zero_page) == &folio->page;
+ return READ_ONCE(huge_zero_folio) == folio;
}
static inline bool is_huge_zero_pmd(pmd_t pmd)
@@ -371,9 +366,14 @@ static inline bool is_huge_zero_pud(pud_t pud)
return false;
}
-struct page *mm_get_huge_zero_page(struct mm_struct *mm);
+struct folio *mm_get_huge_zero_folio(struct mm_struct *mm);
void mm_put_huge_zero_page(struct mm_struct *mm);
+static inline struct page *mm_get_huge_zero_page(struct mm_struct *mm)
+{
+ return &mm_get_huge_zero_folio(mm)->page;
+}
+
#define mk_huge_pmd(page, prot) pmd_mkhuge(mk_pmd(page, prot))
static inline bool thp_migration_supported(void)
@@ -485,11 +485,6 @@ static inline vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
return 0;
}
-static inline bool is_huge_zero_page(const struct page *page)
-{
- return false;
-}
-
static inline bool is_huge_zero_folio(const struct folio *folio)
{
return false;
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 8ee09bfdfdb7..91eb5de3c728 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -74,7 +74,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
struct shrink_control *sc);
static atomic_t huge_zero_refcount;
-struct page *huge_zero_page __read_mostly;
+struct folio *huge_zero_folio __read_mostly;
unsigned long huge_zero_pfn __read_mostly = ~0UL;
unsigned long huge_anon_orders_always __read_mostly;
unsigned long huge_anon_orders_madvise __read_mostly;
@@ -192,24 +192,24 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
static bool get_huge_zero_page(void)
{
- struct page *zero_page;
+ struct folio *zero_folio;
retry:
if (likely(atomic_inc_not_zero(&huge_zero_refcount)))
return true;
- zero_page = alloc_pages((GFP_TRANSHUGE | __GFP_ZERO) & ~__GFP_MOVABLE,
+ zero_folio = folio_alloc((GFP_TRANSHUGE | __GFP_ZERO) & ~__GFP_MOVABLE,
HPAGE_PMD_ORDER);
- if (!zero_page) {
+ if (!zero_folio) {
count_vm_event(THP_ZERO_PAGE_ALLOC_FAILED);
return false;
}
preempt_disable();
- if (cmpxchg(&huge_zero_page, NULL, zero_page)) {
+ if (cmpxchg(&huge_zero_folio, NULL, zero_folio)) {
preempt_enable();
- __free_pages(zero_page, compound_order(zero_page));
+ folio_put(zero_folio);
goto retry;
}
- WRITE_ONCE(huge_zero_pfn, page_to_pfn(zero_page));
+ WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio));
/* We take additional reference here. It will be put back by shrinker */
atomic_set(&huge_zero_refcount, 2);
@@ -227,10 +227,10 @@ static void put_huge_zero_page(void)
BUG_ON(atomic_dec_and_test(&huge_zero_refcount));
}
-struct page *mm_get_huge_zero_page(struct mm_struct *mm)
+struct folio *mm_get_huge_zero_folio(struct mm_struct *mm)
{
if (test_bit(MMF_HUGE_ZERO_PAGE, &mm->flags))
- return READ_ONCE(huge_zero_page);
+ return READ_ONCE(huge_zero_folio);
if (!get_huge_zero_page())
return NULL;
@@ -238,7 +238,7 @@ struct page *mm_get_huge_zero_page(struct mm_struct *mm)
if (test_and_set_bit(MMF_HUGE_ZERO_PAGE, &mm->flags))
put_huge_zero_page();
- return READ_ONCE(huge_zero_page);
+ return READ_ONCE(huge_zero_folio);
}
void mm_put_huge_zero_page(struct mm_struct *mm)
@@ -258,10 +258,10 @@ static unsigned long shrink_huge_zero_page_scan(struct shrinker *shrink,
struct shrink_control *sc)
{
if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) == 1) {
- struct page *zero_page = xchg(&huge_zero_page, NULL);
- BUG_ON(zero_page == NULL);
+ struct folio *zero_folio = xchg(&huge_zero_folio, NULL);
+ BUG_ON(zero_folio == NULL);
WRITE_ONCE(huge_zero_pfn, ~0UL);
- __free_pages(zero_page, compound_order(zero_page));
+ folio_put(zero_folio);
return HPAGE_PMD_NR;
}
@@ -1340,7 +1340,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
* since we already have a zero page to copy. It just takes a
* reference.
*/
- mm_get_huge_zero_page(dst_mm);
+ mm_get_huge_zero_folio(dst_mm);
goto out_zero_page;
}
--
2.43.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 6/8] mm: Convert do_huge_pmd_anonymous_page to huge_zero_folio
2024-03-26 20:28 [PATCH 0/8] Convert huge_zero_page to huge_zero_folio Matthew Wilcox (Oracle)
` (4 preceding siblings ...)
2024-03-26 20:28 ` [PATCH 5/8] mm: Convert huge_zero_page to huge_zero_folio Matthew Wilcox (Oracle)
@ 2024-03-26 20:28 ` Matthew Wilcox (Oracle)
2024-04-04 19:41 ` David Hildenbrand
2024-03-26 20:28 ` [PATCH 7/8] dax: Use huge_zero_folio Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 8/8] mm: Rename mm_put_huge_zero_page to mm_put_huge_zero_folio Matthew Wilcox (Oracle)
7 siblings, 1 reply; 11+ messages in thread
From: Matthew Wilcox (Oracle) @ 2024-03-26 20:28 UTC (permalink / raw
To: Andrew Morton; +Cc: Matthew Wilcox (Oracle), linux-mm
Use folios more widely.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
mm/huge_memory.c | 23 ++++++++++++-----------
1 file changed, 12 insertions(+), 11 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 91eb5de3c728..ff282317e71f 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -971,14 +971,14 @@ gfp_t vma_thp_gfp_mask(struct vm_area_struct *vma)
}
/* Caller must hold page table lock. */
-static void set_huge_zero_page(pgtable_t pgtable, struct mm_struct *mm,
+static void set_huge_zero_folio(pgtable_t pgtable, struct mm_struct *mm,
struct vm_area_struct *vma, unsigned long haddr, pmd_t *pmd,
- struct page *zero_page)
+ struct folio *zero_folio)
{
pmd_t entry;
if (!pmd_none(*pmd))
return;
- entry = mk_pmd(zero_page, vma->vm_page_prot);
+ entry = mk_pmd(&zero_folio->page, vma->vm_page_prot);
entry = pmd_mkhuge(entry);
pgtable_trans_huge_deposit(mm, pmd, pgtable);
set_pmd_at(mm, haddr, pmd, entry);
@@ -1002,13 +1002,14 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
!mm_forbids_zeropage(vma->vm_mm) &&
transparent_hugepage_use_zero_page()) {
pgtable_t pgtable;
- struct page *zero_page;
+ struct folio *zero_folio;
vm_fault_t ret;
+
pgtable = pte_alloc_one(vma->vm_mm);
if (unlikely(!pgtable))
return VM_FAULT_OOM;
- zero_page = mm_get_huge_zero_page(vma->vm_mm);
- if (unlikely(!zero_page)) {
+ zero_folio = mm_get_huge_zero_folio(vma->vm_mm);
+ if (unlikely(!zero_folio)) {
pte_free(vma->vm_mm, pgtable);
count_vm_event(THP_FAULT_FALLBACK);
return VM_FAULT_FALLBACK;
@@ -1026,8 +1027,8 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)
ret = handle_userfault(vmf, VM_UFFD_MISSING);
VM_BUG_ON(ret & VM_FAULT_FALLBACK);
} else {
- set_huge_zero_page(pgtable, vma->vm_mm, vma,
- haddr, vmf->pmd, zero_page);
+ set_huge_zero_folio(pgtable, vma->vm_mm, vma,
+ haddr, vmf->pmd, zero_folio);
update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
spin_unlock(vmf->ptl);
}
@@ -1336,9 +1337,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
*/
if (is_huge_zero_pmd(pmd)) {
/*
- * get_huge_zero_page() will never allocate a new page here,
- * since we already have a zero page to copy. It just takes a
- * reference.
+ * mm_get_huge_zero_folio() will never allocate a new
+ * folio here, since we already have a zero page to
+ * copy. It just takes a reference.
*/
mm_get_huge_zero_folio(dst_mm);
goto out_zero_page;
--
2.43.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 7/8] dax: Use huge_zero_folio
2024-03-26 20:28 [PATCH 0/8] Convert huge_zero_page to huge_zero_folio Matthew Wilcox (Oracle)
` (5 preceding siblings ...)
2024-03-26 20:28 ` [PATCH 6/8] mm: Convert do_huge_pmd_anonymous_page " Matthew Wilcox (Oracle)
@ 2024-03-26 20:28 ` Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 8/8] mm: Rename mm_put_huge_zero_page to mm_put_huge_zero_folio Matthew Wilcox (Oracle)
7 siblings, 0 replies; 11+ messages in thread
From: Matthew Wilcox (Oracle) @ 2024-03-26 20:28 UTC (permalink / raw
To: Andrew Morton; +Cc: Matthew Wilcox (Oracle), linux-mm
Convert from huge_zero_page to huge_zero_folio.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
fs/dax.c | 14 +++++++-------
include/trace/events/fs_dax.h | 16 ++++++++--------
2 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/fs/dax.c b/fs/dax.c
index 423fc1607dfa..becb4a6920c6 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -1207,17 +1207,17 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
struct vm_area_struct *vma = vmf->vma;
struct inode *inode = mapping->host;
pgtable_t pgtable = NULL;
- struct page *zero_page;
+ struct folio *zero_folio;
spinlock_t *ptl;
pmd_t pmd_entry;
pfn_t pfn;
- zero_page = mm_get_huge_zero_page(vmf->vma->vm_mm);
+ zero_folio = mm_get_huge_zero_folio(vmf->vma->vm_mm);
- if (unlikely(!zero_page))
+ if (unlikely(!zero_folio))
goto fallback;
- pfn = page_to_pfn_t(zero_page);
+ pfn = page_to_pfn_t(&zero_folio->page);
*entry = dax_insert_entry(xas, vmf, iter, *entry, pfn,
DAX_PMD | DAX_ZERO_PAGE);
@@ -1237,17 +1237,17 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable);
mm_inc_nr_ptes(vma->vm_mm);
}
- pmd_entry = mk_pmd(zero_page, vmf->vma->vm_page_prot);
+ pmd_entry = mk_pmd(&zero_folio->page, vmf->vma->vm_page_prot);
pmd_entry = pmd_mkhuge(pmd_entry);
set_pmd_at(vmf->vma->vm_mm, pmd_addr, vmf->pmd, pmd_entry);
spin_unlock(ptl);
- trace_dax_pmd_load_hole(inode, vmf, zero_page, *entry);
+ trace_dax_pmd_load_hole(inode, vmf, zero_folio, *entry);
return VM_FAULT_NOPAGE;
fallback:
if (pgtable)
pte_free(vma->vm_mm, pgtable);
- trace_dax_pmd_load_hole_fallback(inode, vmf, zero_page, *entry);
+ trace_dax_pmd_load_hole_fallback(inode, vmf, zero_folio, *entry);
return VM_FAULT_FALLBACK;
}
#else
diff --git a/include/trace/events/fs_dax.h b/include/trace/events/fs_dax.h
index 97b09fcf7e52..86fe6aecff1e 100644
--- a/include/trace/events/fs_dax.h
+++ b/include/trace/events/fs_dax.h
@@ -62,14 +62,14 @@ DEFINE_PMD_FAULT_EVENT(dax_pmd_fault_done);
DECLARE_EVENT_CLASS(dax_pmd_load_hole_class,
TP_PROTO(struct inode *inode, struct vm_fault *vmf,
- struct page *zero_page,
+ struct folio *zero_folio,
void *radix_entry),
- TP_ARGS(inode, vmf, zero_page, radix_entry),
+ TP_ARGS(inode, vmf, zero_folio, radix_entry),
TP_STRUCT__entry(
__field(unsigned long, ino)
__field(unsigned long, vm_flags)
__field(unsigned long, address)
- __field(struct page *, zero_page)
+ __field(struct folio *, zero_folio)
__field(void *, radix_entry)
__field(dev_t, dev)
),
@@ -78,17 +78,17 @@ DECLARE_EVENT_CLASS(dax_pmd_load_hole_class,
__entry->ino = inode->i_ino;
__entry->vm_flags = vmf->vma->vm_flags;
__entry->address = vmf->address;
- __entry->zero_page = zero_page;
+ __entry->zero_folio = zero_folio;
__entry->radix_entry = radix_entry;
),
- TP_printk("dev %d:%d ino %#lx %s address %#lx zero_page %p "
+ TP_printk("dev %d:%d ino %#lx %s address %#lx zero_folio %p "
"radix_entry %#lx",
MAJOR(__entry->dev),
MINOR(__entry->dev),
__entry->ino,
__entry->vm_flags & VM_SHARED ? "shared" : "private",
__entry->address,
- __entry->zero_page,
+ __entry->zero_folio,
(unsigned long)__entry->radix_entry
)
)
@@ -96,8 +96,8 @@ DECLARE_EVENT_CLASS(dax_pmd_load_hole_class,
#define DEFINE_PMD_LOAD_HOLE_EVENT(name) \
DEFINE_EVENT(dax_pmd_load_hole_class, name, \
TP_PROTO(struct inode *inode, struct vm_fault *vmf, \
- struct page *zero_page, void *radix_entry), \
- TP_ARGS(inode, vmf, zero_page, radix_entry))
+ struct folio *zero_folio, void *radix_entry), \
+ TP_ARGS(inode, vmf, zero_folio, radix_entry))
DEFINE_PMD_LOAD_HOLE_EVENT(dax_pmd_load_hole);
DEFINE_PMD_LOAD_HOLE_EVENT(dax_pmd_load_hole_fallback);
--
2.43.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 8/8] mm: Rename mm_put_huge_zero_page to mm_put_huge_zero_folio
2024-03-26 20:28 [PATCH 0/8] Convert huge_zero_page to huge_zero_folio Matthew Wilcox (Oracle)
` (6 preceding siblings ...)
2024-03-26 20:28 ` [PATCH 7/8] dax: Use huge_zero_folio Matthew Wilcox (Oracle)
@ 2024-03-26 20:28 ` Matthew Wilcox (Oracle)
7 siblings, 0 replies; 11+ messages in thread
From: Matthew Wilcox (Oracle) @ 2024-03-26 20:28 UTC (permalink / raw
To: Andrew Morton; +Cc: Matthew Wilcox (Oracle), linux-mm
Also remove mm_get_huge_zero_page() now it has no users.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
include/linux/huge_mm.h | 9 ++-------
kernel/fork.c | 2 +-
mm/huge_memory.c | 2 +-
3 files changed, 4 insertions(+), 9 deletions(-)
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 7ba59ba36354..9d11e886aa9a 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -367,12 +367,7 @@ static inline bool is_huge_zero_pud(pud_t pud)
}
struct folio *mm_get_huge_zero_folio(struct mm_struct *mm);
-void mm_put_huge_zero_page(struct mm_struct *mm);
-
-static inline struct page *mm_get_huge_zero_page(struct mm_struct *mm)
-{
- return &mm_get_huge_zero_folio(mm)->page;
-}
+void mm_put_huge_zero_folio(struct mm_struct *mm);
#define mk_huge_pmd(page, prot) pmd_mkhuge(mk_pmd(page, prot))
@@ -500,7 +495,7 @@ static inline bool is_huge_zero_pud(pud_t pud)
return false;
}
-static inline void mm_put_huge_zero_page(struct mm_struct *mm)
+static inline void mm_put_huge_zero_folio(struct mm_struct *mm)
{
return;
}
diff --git a/kernel/fork.c b/kernel/fork.c
index 39a5046c2f0b..84de5faa8c9a 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -1343,7 +1343,7 @@ static inline void __mmput(struct mm_struct *mm)
ksm_exit(mm);
khugepaged_exit(mm); /* must run before exit_mmap */
exit_mmap(mm);
- mm_put_huge_zero_page(mm);
+ mm_put_huge_zero_folio(mm);
set_mm_exe_file(mm, NULL);
if (!list_empty(&mm->mmlist)) {
spin_lock(&mmlist_lock);
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index ff282317e71f..b49fcb8a16cc 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -241,7 +241,7 @@ struct folio *mm_get_huge_zero_folio(struct mm_struct *mm)
return READ_ONCE(huge_zero_folio);
}
-void mm_put_huge_zero_page(struct mm_struct *mm)
+void mm_put_huge_zero_folio(struct mm_struct *mm)
{
if (test_bit(MMF_HUGE_ZERO_PAGE, &mm->flags))
put_huge_zero_page();
--
2.43.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 3/8] mm: Add pmd_folio()
2024-03-26 20:28 ` [PATCH 3/8] mm: Add pmd_folio() Matthew Wilcox (Oracle)
@ 2024-04-04 19:39 ` David Hildenbrand
0 siblings, 0 replies; 11+ messages in thread
From: David Hildenbrand @ 2024-04-04 19:39 UTC (permalink / raw
To: Matthew Wilcox (Oracle), Andrew Morton; +Cc: linux-mm
On 26.03.24 21:28, Matthew Wilcox (Oracle) wrote:
> Convert directly from a pmd to a folio without going through another
> representation first. For now this is just a slightly shorter way to
> write it, but it might end up being more efficient later.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> ---
> include/linux/pgtable.h | 2 ++
> mm/huge_memory.c | 6 +++---
> mm/madvise.c | 2 +-
> mm/mempolicy.c | 2 +-
> mm/mlock.c | 2 +-
> mm/userfaultfd.c | 2 +-
> 6 files changed, 9 insertions(+), 7 deletions(-)
>
> diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
> index 600e17d03659..09c85c7bf9c2 100644
> --- a/include/linux/pgtable.h
> +++ b/include/linux/pgtable.h
> @@ -50,6 +50,8 @@
> #define pmd_pgtable(pmd) pmd_page(pmd)
> #endif
>
> +#define pmd_folio(pmd) page_folio(pmd_page(pmd))
Heh, I even used that on one of my recent patches and was sad not to
find pud_folio(). ;)
Reviewed-by: David Hildenbrand <david@redhat.com>
--
Cheers,
David / dhildenb
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 6/8] mm: Convert do_huge_pmd_anonymous_page to huge_zero_folio
2024-03-26 20:28 ` [PATCH 6/8] mm: Convert do_huge_pmd_anonymous_page " Matthew Wilcox (Oracle)
@ 2024-04-04 19:41 ` David Hildenbrand
0 siblings, 0 replies; 11+ messages in thread
From: David Hildenbrand @ 2024-04-04 19:41 UTC (permalink / raw
To: Matthew Wilcox (Oracle), Andrew Morton; +Cc: linux-mm
On 26.03.24 21:28, Matthew Wilcox (Oracle) wrote:
> Use folios more widely.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> ---
Reviewed-by: David Hildenbrand <david@redhat.com>
--
Cheers,
David / dhildenb
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-04-04 19:41 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-03-26 20:28 [PATCH 0/8] Convert huge_zero_page to huge_zero_folio Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 1/8] sparc: Use is_huge_zero_pmd() Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 2/8] mm: Add is_huge_zero_folio() Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 3/8] mm: Add pmd_folio() Matthew Wilcox (Oracle)
2024-04-04 19:39 ` David Hildenbrand
2024-03-26 20:28 ` [PATCH 4/8] mm: Convert migrate_vma_collect_pmd to use a folio Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 5/8] mm: Convert huge_zero_page to huge_zero_folio Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 6/8] mm: Convert do_huge_pmd_anonymous_page " Matthew Wilcox (Oracle)
2024-04-04 19:41 ` David Hildenbrand
2024-03-26 20:28 ` [PATCH 7/8] dax: Use huge_zero_folio Matthew Wilcox (Oracle)
2024-03-26 20:28 ` [PATCH 8/8] mm: Rename mm_put_huge_zero_page to mm_put_huge_zero_folio Matthew Wilcox (Oracle)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).