Commit f728b9c4 authored by John Hubbard's avatar John Hubbard Committed by Linus Torvalds
Browse files

mm: change lookup_node() to use get_user_pages_fast() 

The purpose of calling get_user_pages_locked() from lookup_node() was to
allow for unlocking the mmap_lock when reading a page from the disk
during a page fault (hidden behind VM_FAULT_RETRY).  The idea was to
reduce contention on the heavily-used mmap_lock.  (Thanks to Jan Kara
for clearly pointing that out, and in fact I've used some of his wording
here.)

However, it is unlikely for lookup_node() to take a page fault.  With
that in mind, change over to calling get_user_pages_fast().  This
simplifies the code, runs a little faster in the expected case, and
allows removing get_user_pages_locked() entirely, in a subsequent patch.

Link: https://lkml.kernel.org/r/20220204020010.68930-5-jhubbard@nvidia.com


Signed-off-by: default avatarJohn Hubbard <jhubbard@nvidia.com>
Reviewed-by: default avatarJan Kara <jack@suse.cz>
Reviewed-by: default avatarJason Gunthorpe <jgg@nvidia.com>
Reviewed-by: default avatarClaudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Lukas Bulwahn <lukas.bulwahn@gmail.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Peter Xu <peterx@redhat.com>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent ad6c4412

mm/mempolicy.c

+9 −12
Original line number Diff line number Diff line
@@ -907,17 +907,14 @@ static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes) 
static int lookup_node(struct mm_struct *mm, unsigned long addr)

{

	struct page *p = NULL;

	int err;

	int ret;



	int locked = 1;

	err = get_user_pages_locked(addr & PAGE_MASK, 1, 0, &p, &locked);

	if (err > 0) {

		err = page_to_nid(p);

	ret = get_user_pages_fast(addr & PAGE_MASK, 1, 0, &p);

	if (ret > 0) {

		ret = page_to_nid(p);

		put_page(p);

	}

	if (locked)

		mmap_read_unlock(mm);

	return err;

	return ret;

}



/* Retrieve NUMA policy */
@@ -968,14 +965,14 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, 
	if (flags & MPOL_F_NODE) {

		if (flags & MPOL_F_ADDR) {

			/*

			 * Take a refcount on the mpol, lookup_node()

			 * will drop the mmap_lock, so after calling

			 * lookup_node() only "pol" remains valid, "vma"

			 * is stale.

			 * Take a refcount on the mpol, because we are about to

			 * drop the mmap_lock, after which only "pol" remains

			 * valid, "vma" is stale.

			 */

			pol_refcount = pol;

			vma = NULL;

			mpol_get(pol);

			mmap_read_unlock(mm);

			err = lookup_node(mm, addr);

			if (err < 0)

				goto out;