Unverified Commit 542c5d6d authored by openeuler-ci-bot's avatar openeuler-ci-bot Committed by Gitee
Browse files

!5613 mm: backport rmap interface overhaul 

Merge Pull Request from: @ci-robot 
 
PR sync from: Kefeng Wang <wangkefeng.wang@huawei.com>
https://mailweb.openeuler.org/hyperkitty/list/kernel@openeuler.org/message/PERFLGWYSWOVROSJC6CW7FLUTNR7OWTP/ 
The rmap interface overhaul is to speeds up the folio operations.

David Hildenbrand (43):
  mm/rmap: rename hugepage_add* to hugetlb_add*
  mm/rmap: introduce and use hugetlb_remove_rmap()
  mm/rmap: introduce and use hugetlb_add_file_rmap()
  mm/rmap: introduce and use hugetlb_try_dup_anon_rmap()
  mm/rmap: introduce and use hugetlb_try_share_anon_rmap()
  mm/rmap: add hugetlb sanity checks for anon rmap handling
  mm/rmap: convert folio_add_file_rmap_range() into
    folio_add_file_rmap_[pte|ptes|pmd]()
  mm/memory: page_add_file_rmap() -> folio_add_file_rmap_[pte|pmd]()
  mm/huge_memory: page_add_file_rmap() -> folio_add_file_rmap_pmd()
  mm/migrate: page_add_file_rmap() -> folio_add_file_rmap_pte()
  mm/userfaultfd: page_add_file_rmap() -> folio_add_file_rmap_pte()
  mm/rmap: remove page_add_file_rmap()
  mm/rmap: factor out adding folio mappings into __folio_add_rmap()
  mm/rmap: introduce folio_add_anon_rmap_[pte|ptes|pmd]()
  mm/huge_memory: batch rmap operations in __split_huge_pmd_locked()
  mm/huge_memory: page_add_anon_rmap() -> folio_add_anon_rmap_pmd()
  mm/migrate: page_add_anon_rmap() -> folio_add_anon_rmap_pte()
  mm/ksm: page_add_anon_rmap() -> folio_add_anon_rmap_pte()
  mm/swapfile: page_add_anon_rmap() -> folio_add_anon_rmap_pte()
  mm/memory: page_add_anon_rmap() -> folio_add_anon_rmap_pte()
  mm/rmap: remove page_add_anon_rmap()
  mm/rmap: remove RMAP_COMPOUND
  mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()
  kernel/events/uprobes: page_remove_rmap() -> folio_remove_rmap_pte()
  mm/huge_memory: page_remove_rmap() -> folio_remove_rmap_pmd()
  mm/khugepaged: page_remove_rmap() -> folio_remove_rmap_pte()
  mm/ksm: page_remove_rmap() -> folio_remove_rmap_pte()
  mm/memory: page_remove_rmap() -> folio_remove_rmap_pte()
  mm/migrate_device: page_remove_rmap() -> folio_remove_rmap_pte()
  mm/rmap: page_remove_rmap() -> folio_remove_rmap_pte()
  Documentation: stop referring to page_remove_rmap()
  mm/rmap: remove page_remove_rmap()
  mm/rmap: convert page_dup_file_rmap() to
    folio_dup_file_rmap_[pte|ptes|pmd]()
  mm/rmap: introduce folio_try_dup_anon_rmap_[pte|ptes|pmd]()
  mm/huge_memory: page_try_dup_anon_rmap() ->
    folio_try_dup_anon_rmap_pmd()
  mm/memory: page_try_dup_anon_rmap() -> folio_try_dup_anon_rmap_pte()
  mm/rmap: remove page_try_dup_anon_rmap()
  mm: convert page_try_share_anon_rmap() to
    folio_try_share_anon_rmap_[pte|pmd]()
  mm/rmap: rename COMPOUND_MAPPED to ENTIRELY_MAPPED
  mm: remove one last reference to page_add_*_rmap()
  mm/rmap: silence VM_WARN_ON_FOLIO() in __folio_rmap_sanity_checks()
  mm/huge_memory: fix folio_set_dirty() vs. folio_mark_dirty()
  mm/memory: fix folio_set_dirty() vs. folio_mark_dirty() in
    zap_pte_range()

Kefeng Wang (1):
  mm: userswap: page_remove_rmap() -> folio_remove_rmap_pte()

Vishal Moola (Oracle) (5):
  mm/khugepaged: convert __collapse_huge_page_isolate() to use folios
  mm/khugepaged: convert hpage_collapse_scan_pmd() to use folios
  mm/khugepaged: convert is_refcount_suitable() to use folios
  mm/khugepaged: convert alloc_charge_hpage() to use folios
  mm/khugepaged: convert collapse_pte_mapped_thp() to use folios


-- 
2.27.0
 
https://gitee.com/openeuler/kernel/issues/I9CESE 
 
Link:https://gitee.com/openeuler/kernel/pulls/5613

 

Reviewed-by: default avatarWeilong Chen <chenweilong@huawei.com>
Reviewed-by: default avatarXu Kuohai <xukuohai@huawei.com>
Reviewed-by: default avatarLu Jialin <lujialin4@huawei.com>
Signed-off-by: default avatarZheng Zengkai <zhengzengkai@huawei.com>
parents dc68fe80 55d1f5a4

Documentation/mm/transhuge.rst

+2 −2
Original line number Diff line number Diff line
@@ -117,7 +117,7 @@ pages: 


  - map/unmap of a PMD entry for the whole THP increment/decrement

    folio->_entire_mapcount and also increment/decrement

    folio->_nr_pages_mapped by COMPOUND_MAPPED when _entire_mapcount

    folio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount

    goes from -1 to 0 or 0 to -1.



  - map/unmap of individual pages with PTE entry increment/decrement
@@ -156,7 +156,7 @@ Partial unmap and deferred_split_folio() 


Unmapping part of THP (with munmap() or other way) is not going to free

memory immediately. Instead, we detect that a subpage of THP is not in use

in page_remove_rmap() and queue the THP for splitting if memory pressure

in folio_remove_rmap_*() and queue the THP for splitting if memory pressure

comes. Splitting will free up unused subpages.



Splitting the page right away is not an option due to locking context in

Documentation/mm/unevictable-lru.rst

+2 −2
Original line number Diff line number Diff line
@@ -486,7 +486,7 @@ munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. 
Before the unevictable/mlock changes, mlocking did not mark the pages in any

way, so unmapping them required no processing.



For each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls

For each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls

munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED

(unless it was a PTE mapping of a part of a transparent huge page).
@@ -511,7 +511,7 @@ userspace; truncation even unmaps and deletes any private anonymous pages 
which had been Copied-On-Write from the file pages now being truncated.



Mlocked pages can be munlocked and deleted in this way: like with munmap(),

for each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls

for each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls

munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED

(unless it was a PTE mapping of a part of a transparent huge page).

include/linux/memcontrol.h

+0 −14
Original line number Diff line number Diff line
@@ -1213,15 +1213,6 @@ static inline void count_memcg_events(struct mem_cgroup *memcg, 
	local_irq_restore(flags);

}



static inline void count_memcg_page_event(struct page *page,

					  enum vm_event_item idx)

{

	struct mem_cgroup *memcg = page_memcg(page);



	if (memcg)

		count_memcg_events(memcg, idx, 1);

}



static inline void count_memcg_folio_events(struct folio *folio,

		enum vm_event_item idx, unsigned long nr)

{
@@ -1741,11 +1732,6 @@ static inline void __count_memcg_events(struct mem_cgroup *memcg, 
{

}



static inline void count_memcg_page_event(struct page *page,

					  int idx)

{

}



static inline void count_memcg_folio_events(struct folio *folio,

		enum vm_event_item idx, unsigned long nr)

{

include/linux/mm.h

+3 −3
Original line number Diff line number Diff line
@@ -1980,15 +1980,15 @@ static inline bool page_maybe_dma_pinned(struct page *page) 
 *

 * The caller has to hold the PT lock and the vma->vm_mm->->write_protect_seq.

 */

static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma,

					  struct page *page)

static inline bool folio_needs_cow_for_dma(struct vm_area_struct *vma,

					  struct folio *folio)

{

	VM_BUG_ON(!(raw_read_seqcount(&vma->vm_mm->write_protect_seq) & 1));



	if (!test_bit(MMF_HAS_PINNED, &vma->vm_mm->flags))

		return false;



	return page_maybe_dma_pinned(page);

	return folio_maybe_dma_pinned(folio);

}



/**

include/linux/rmap.h

+329 −75
Original line number Diff line number Diff line
@@ -172,133 +172,325 @@ struct anon_vma *folio_get_anon_vma(struct folio *folio); 
typedef int __bitwise rmap_t;



/*

 * No special request: if the page is a subpage of a compound page, it is

 * mapped via a PTE. The mapped (sub)page is possibly shared between processes.

 * No special request: A mapped anonymous (sub)page is possibly shared between

 * processes.

 */

#define RMAP_NONE		((__force rmap_t)0)



/* The (sub)page is exclusive to a single process. */

/* The anonymous (sub)page is exclusive to a single process. */

#define RMAP_EXCLUSIVE		((__force rmap_t)BIT(0))



/*

 * The compound page is not mapped via PTEs, but instead via a single PMD and

 * should be accounted accordingly.

 * Internally, we're using an enum to specify the granularity. We make the

 * compiler emit specialized code for each granularity.

 */

#define RMAP_COMPOUND		((__force rmap_t)BIT(1))

enum rmap_level {

	RMAP_LEVEL_PTE = 0,

	RMAP_LEVEL_PMD,

};



static inline void __folio_rmap_sanity_checks(struct folio *folio,

		struct page *page, int nr_pages, enum rmap_level level)

{

	/* hugetlb folios are handled separately. */

	VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio);



	/*

	 * TODO: we get driver-allocated folios that have nothing to do with

	 * the rmap using vm_insert_page(); therefore, we cannot assume that

	 * folio_test_large_rmappable() holds for large folios. We should

	 * handle any desired mapcount+stats accounting for these folios in

	 * VM_MIXEDMAP VMAs separately, and then sanity-check here that

	 * we really only get rmappable folios.

	 */



	VM_WARN_ON_ONCE(nr_pages <= 0);

	VM_WARN_ON_FOLIO(page_folio(page) != folio, folio);

	VM_WARN_ON_FOLIO(page_folio(page + nr_pages - 1) != folio, folio);



	switch (level) {

	case RMAP_LEVEL_PTE:

		break;

	case RMAP_LEVEL_PMD:

		/*

		 * We don't support folios larger than a single PMD yet. So

		 * when RMAP_LEVEL_PMD is set, we assume that we are creating

		 * a single "entire" mapping of the folio.

		 */

		VM_WARN_ON_FOLIO(folio_nr_pages(folio) != HPAGE_PMD_NR, folio);

		VM_WARN_ON_FOLIO(nr_pages != HPAGE_PMD_NR, folio);

		break;

	default:

		VM_WARN_ON_ONCE(true);

	}

}



/*

 * rmap interfaces called when adding or removing pte of page

 */

void folio_move_anon_rmap(struct folio *, struct vm_area_struct *);

void page_add_anon_rmap(struct page *, struct vm_area_struct *,

		unsigned long address, rmap_t flags);

void folio_add_anon_rmap_ptes(struct folio *, struct page *, int nr_pages,

		struct vm_area_struct *, unsigned long address, rmap_t flags);

#define folio_add_anon_rmap_pte(folio, page, vma, address, flags) \

	folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags)

void folio_add_anon_rmap_pmd(struct folio *, struct page *,

		struct vm_area_struct *, unsigned long address, rmap_t flags);

void page_add_new_anon_rmap(struct page *, struct vm_area_struct *,

		unsigned long address);

void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *,

		unsigned long address);

void page_add_file_rmap(struct page *, struct vm_area_struct *,

		bool compound);

void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr,

		struct vm_area_struct *, bool compound);

void page_remove_rmap(struct page *, struct vm_area_struct *,

		bool compound);



void hugepage_add_anon_rmap(struct folio *, struct vm_area_struct *,

void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages,

		struct vm_area_struct *);

#define folio_add_file_rmap_pte(folio, page, vma) \

	folio_add_file_rmap_ptes(folio, page, 1, vma)

void folio_add_file_rmap_pmd(struct folio *, struct page *,

		struct vm_area_struct *);

void folio_remove_rmap_ptes(struct folio *, struct page *, int nr_pages,

		struct vm_area_struct *);

#define folio_remove_rmap_pte(folio, page, vma) \

	folio_remove_rmap_ptes(folio, page, 1, vma)

void folio_remove_rmap_pmd(struct folio *, struct page *,

		struct vm_area_struct *);



void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *,

		unsigned long address, rmap_t flags);

void hugepage_add_new_anon_rmap(struct folio *, struct vm_area_struct *,

void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *,

		unsigned long address);



static inline void __page_dup_rmap(struct page *page, bool compound)

/* See folio_try_dup_anon_rmap_*() */

static inline int hugetlb_try_dup_anon_rmap(struct folio *folio,

		struct vm_area_struct *vma)

{

	if (compound) {

		struct folio *folio = (struct folio *)page;

	VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio);

	VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);



		VM_BUG_ON_PAGE(compound && !PageHead(page), page);

	if (PageAnonExclusive(&folio->page)) {

		if (unlikely(folio_needs_cow_for_dma(vma, folio)))

			return -EBUSY;

		ClearPageAnonExclusive(&folio->page);

	}

	atomic_inc(&folio->_entire_mapcount);

	} else {

		atomic_inc(&page->_mapcount);

	return 0;

}



/* See folio_try_share_anon_rmap_*() */

static inline int hugetlb_try_share_anon_rmap(struct folio *folio)

{

	VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio);

	VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);

	VM_WARN_ON_FOLIO(!PageAnonExclusive(&folio->page), folio);



	/* Paired with the memory barrier in try_grab_folio(). */

	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP))

		smp_mb();



	if (unlikely(folio_maybe_dma_pinned(folio)))

		return -EBUSY;

	ClearPageAnonExclusive(&folio->page);



	/*

	 * This is conceptually a smp_wmb() paired with the smp_rmb() in

	 * gup_must_unshare().

	 */

	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP))

		smp_mb__after_atomic();

	return 0;

}



static inline void hugetlb_add_file_rmap(struct folio *folio)

{

	VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio);

	VM_WARN_ON_FOLIO(folio_test_anon(folio), folio);



	atomic_inc(&folio->_entire_mapcount);

}



static inline void page_dup_file_rmap(struct page *page, bool compound)

static inline void hugetlb_remove_rmap(struct folio *folio)

{

	__page_dup_rmap(page, compound);

	VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio);



	atomic_dec(&folio->_entire_mapcount);

}



static __always_inline void __folio_dup_file_rmap(struct folio *folio,

		struct page *page, int nr_pages, enum rmap_level level)

{

	__folio_rmap_sanity_checks(folio, page, nr_pages, level);



	switch (level) {

	case RMAP_LEVEL_PTE:

		do {

			atomic_inc(&page->_mapcount);

		} while (page++, --nr_pages > 0);

		break;

	case RMAP_LEVEL_PMD:

		atomic_inc(&folio->_entire_mapcount);

		break;

	}

}



/**

 * page_try_dup_anon_rmap - try duplicating a mapping of an already mapped

 *			    anonymous page

 * @page: the page to duplicate the mapping for

 * @compound: the page is mapped as compound or as a small page

 * @vma: the source vma

 * folio_dup_file_rmap_ptes - duplicate PTE mappings of a page range of a folio

 * @folio:	The folio to duplicate the mappings of

 * @page:	The first page to duplicate the mappings of

 * @nr_pages:	The number of pages of which the mapping will be duplicated

 *

 * The caller needs to hold the PT lock and the vma->vma_mm->write_protect_seq.

 * The page range of the folio is defined by [page, page + nr_pages)

 *

 * Duplicating the mapping can only fail if the page may be pinned; device

 * private pages cannot get pinned and consequently this function cannot fail.

 * The caller needs to hold the page table lock.

 */

static inline void folio_dup_file_rmap_ptes(struct folio *folio,

		struct page *page, int nr_pages)

{

	__folio_dup_file_rmap(folio, page, nr_pages, RMAP_LEVEL_PTE);

}

#define folio_dup_file_rmap_pte(folio, page) \

	folio_dup_file_rmap_ptes(folio, page, 1)



/**

 * folio_dup_file_rmap_pmd - duplicate a PMD mapping of a page range of a folio

 * @folio:	The folio to duplicate the mapping of

 * @page:	The first page to duplicate the mapping of

 *

 * If duplicating the mapping succeeds, the page has to be mapped R/O into

 * the parent and the child. It must *not* get mapped writable after this call.

 * The page range of the folio is defined by [page, page + HPAGE_PMD_NR)

 *

 * Returns 0 if duplicating the mapping succeeded. Returns -EBUSY otherwise.

 * The caller needs to hold the page table lock.

 */

static inline int page_try_dup_anon_rmap(struct page *page, bool compound,

					 struct vm_area_struct *vma)

static inline void folio_dup_file_rmap_pmd(struct folio *folio,

		struct page *page)

{

	VM_BUG_ON_PAGE(!PageAnon(page), page);

#ifdef CONFIG_TRANSPARENT_HUGEPAGE

	__folio_dup_file_rmap(folio, page, HPAGE_PMD_NR, RMAP_LEVEL_PTE);

#else

	WARN_ON_ONCE(true);

#endif

}



	/*

	 * No need to check+clear for already shared pages, including KSM

	 * pages.

	 */

	if (!PageAnonExclusive(page))

		goto dup;

static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio,

		struct page *page, int nr_pages, struct vm_area_struct *src_vma,

		enum rmap_level level)

{

	bool maybe_pinned;

	int i;



	VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);

	__folio_rmap_sanity_checks(folio, page, nr_pages, level);



	/*

	 * If this page may have been pinned by the parent process,

	 * don't allow to duplicate the mapping but instead require to e.g.,

	 * copy the page immediately for the child so that we'll always

	 * guarantee the pinned page won't be randomly replaced in the

	 * If this folio may have been pinned by the parent process,

	 * don't allow to duplicate the mappings but instead require to e.g.,

	 * copy the subpage immediately for the child so that we'll always

	 * guarantee the pinned folio won't be randomly replaced in the

	 * future on write faults.

	 */

	if (likely(!is_device_private_page(page)) &&

	    unlikely(page_needs_cow_for_dma(vma, page)))

		return -EBUSY;

	maybe_pinned = likely(!folio_is_device_private(folio)) &&

		       unlikely(folio_needs_cow_for_dma(src_vma, folio));



	ClearPageAnonExclusive(page);

	/*

	 * It's okay to share the anon page between both processes, mapping

	 * the page R/O into both processes.

	 * No need to check+clear for already shared PTEs/PMDs of the

	 * folio. But if any page is PageAnonExclusive, we must fallback to

	 * copying if the folio maybe pinned.

	 */

dup:

	__page_dup_rmap(page, compound);

	switch (level) {

	case RMAP_LEVEL_PTE:

		if (unlikely(maybe_pinned)) {

			for (i = 0; i < nr_pages; i++)

				if (PageAnonExclusive(page + i))

					return -EBUSY;

		}

		do {

			if (PageAnonExclusive(page))

				ClearPageAnonExclusive(page);

			atomic_inc(&page->_mapcount);

		} while (page++, --nr_pages > 0);

		break;

	case RMAP_LEVEL_PMD:

		if (PageAnonExclusive(page)) {

			if (unlikely(maybe_pinned))

				return -EBUSY;

			ClearPageAnonExclusive(page);

		}

		atomic_inc(&folio->_entire_mapcount);

		break;

	}

	return 0;

}



/**

 * page_try_share_anon_rmap - try marking an exclusive anonymous page possibly

 *			      shared to prepare for KSM or temporary unmapping

 * @page: the exclusive anonymous page to try marking possibly shared

 * folio_try_dup_anon_rmap_ptes - try duplicating PTE mappings of a page range

 *				  of a folio

 * @folio:	The folio to duplicate the mappings of

 * @page:	The first page to duplicate the mappings of

 * @nr_pages:	The number of pages of which the mapping will be duplicated

 * @src_vma:	The vm area from which the mappings are duplicated

 *

 * The page range of the folio is defined by [page, page + nr_pages)

 *

 * The caller needs to hold the page table lock and the

 * vma->vma_mm->write_protect_seq.

 *

 * Duplicating the mappings can only fail if the folio may be pinned; device

 * private folios cannot get pinned and consequently this function cannot fail

 * for them.

 *

 * If duplicating the mappings succeeded, the duplicated PTEs have to be R/O in

 * the parent and the child. They must *not* be writable after this call

 * succeeded.

 *

 * Returns 0 if duplicating the mappings succeeded. Returns -EBUSY otherwise.

 */

static inline int folio_try_dup_anon_rmap_ptes(struct folio *folio,

		struct page *page, int nr_pages, struct vm_area_struct *src_vma)

{

	return __folio_try_dup_anon_rmap(folio, page, nr_pages, src_vma,

					 RMAP_LEVEL_PTE);

}

#define folio_try_dup_anon_rmap_pte(folio, page, vma) \

	folio_try_dup_anon_rmap_ptes(folio, page, 1, vma)



/**

 * folio_try_dup_anon_rmap_pmd - try duplicating a PMD mapping of a page range

 *				 of a folio

 * @folio:	The folio to duplicate the mapping of

 * @page:	The first page to duplicate the mapping of

 * @src_vma:	The vm area from which the mapping is duplicated

 *

 * The page range of the folio is defined by [page, page + HPAGE_PMD_NR)

 *

 * The caller needs to hold the PT lock and has to have the page table entry

 * cleared/invalidated.

 * The caller needs to hold the page table lock and the

 * vma->vma_mm->write_protect_seq.

 *

 * This is similar to page_try_dup_anon_rmap(), however, not used during fork()

 * to duplicate a mapping, but instead to prepare for KSM or temporarily

 * unmapping a page (swap, migration) via page_remove_rmap().

 * Duplicating the mapping can only fail if the folio may be pinned; device

 * private folios cannot get pinned and consequently this function cannot fail

 * for them.

 *

 * Marking the page shared can only fail if the page may be pinned; device

 * private pages cannot get pinned and consequently this function cannot fail.

 * If duplicating the mapping succeeds, the duplicated PMD has to be R/O in

 * the parent and the child. They must *not* be writable after this call

 * succeeded.

 *

 * Returns 0 if marking the page possibly shared succeeded. Returns -EBUSY

 * otherwise.

 * Returns 0 if duplicating the mapping succeeded. Returns -EBUSY otherwise.

 */

static inline int page_try_share_anon_rmap(struct page *page)

static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio,

		struct page *page, struct vm_area_struct *src_vma)

{

	VM_BUG_ON_PAGE(!PageAnon(page) || !PageAnonExclusive(page), page);

#ifdef CONFIG_TRANSPARENT_HUGEPAGE

	return __folio_try_dup_anon_rmap(folio, page, HPAGE_PMD_NR, src_vma,

					 RMAP_LEVEL_PMD);

#else

	WARN_ON_ONCE(true);

	return -EBUSY;

#endif

}



	/* device private pages cannot get pinned via GUP. */

	if (unlikely(is_device_private_page(page))) {

static __always_inline int __folio_try_share_anon_rmap(struct folio *folio,

		struct page *page, int nr_pages, enum rmap_level level)

{

	VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);

	VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio);

	__folio_rmap_sanity_checks(folio, page, nr_pages, level);



	/* device private folios cannot get pinned via GUP. */

	if (unlikely(folio_is_device_private(folio))) {

		ClearPageAnonExclusive(page);

		return 0;

	}
@@ -349,7 +541,7 @@ static inline int page_try_share_anon_rmap(struct page *page) 
	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP))

		smp_mb();



	if (unlikely(page_maybe_dma_pinned(page)))

	if (unlikely(folio_maybe_dma_pinned(folio)))

		return -EBUSY;

	ClearPageAnonExclusive(page);
@@ -362,6 +554,68 @@ static inline int page_try_share_anon_rmap(struct page *page) 
	return 0;

}



/**

 * folio_try_share_anon_rmap_pte - try marking an exclusive anonymous page

 *				   mapped by a PTE possibly shared to prepare

 *				   for KSM or temporary unmapping

 * @folio:	The folio to share a mapping of

 * @page:	The mapped exclusive page

 *

 * The caller needs to hold the page table lock and has to have the page table

 * entries cleared/invalidated.

 *

 * This is similar to folio_try_dup_anon_rmap_pte(), however, not used during

 * fork() to duplicate mappings, but instead to prepare for KSM or temporarily

 * unmapping parts of a folio (swap, migration) via folio_remove_rmap_pte().

 *

 * Marking the mapped page shared can only fail if the folio maybe pinned;

 * device private folios cannot get pinned and consequently this function cannot

 * fail.

 *

 * Returns 0 if marking the mapped page possibly shared succeeded. Returns

 * -EBUSY otherwise.

 */

static inline int folio_try_share_anon_rmap_pte(struct folio *folio,

		struct page *page)

{

	return __folio_try_share_anon_rmap(folio, page, 1, RMAP_LEVEL_PTE);

}



/**

 * folio_try_share_anon_rmap_pmd - try marking an exclusive anonymous page

 *				   range mapped by a PMD possibly shared to

 *				   prepare for temporary unmapping

 * @folio:	The folio to share the mapping of

 * @page:	The first page to share the mapping of

 *

 * The page range of the folio is defined by [page, page + HPAGE_PMD_NR)

 *

 * The caller needs to hold the page table lock and has to have the page table

 * entries cleared/invalidated.

 *

 * This is similar to folio_try_dup_anon_rmap_pmd(), however, not used during

 * fork() to duplicate a mapping, but instead to prepare for temporarily

 * unmapping parts of a folio (swap, migration) via folio_remove_rmap_pmd().

 *

 * Marking the mapped pages shared can only fail if the folio maybe pinned;

 * device private folios cannot get pinned and consequently this function cannot

 * fail.

 *

 * Returns 0 if marking the mapped pages possibly shared succeeded. Returns

 * -EBUSY otherwise.

 */

static inline int folio_try_share_anon_rmap_pmd(struct folio *folio,

		struct page *page)

{

#ifdef CONFIG_TRANSPARENT_HUGEPAGE

	return __folio_try_share_anon_rmap(folio, page, HPAGE_PMD_NR,

					   RMAP_LEVEL_PMD);

#else

	WARN_ON_ONCE(true);

	return -EBUSY;

#endif

}



/*

 * Called from mm/vmscan.c to handle paging out

 */