Commit ce76f9a1 authored by Jesper Dangaard Brouer's avatar Jesper Dangaard Brouer Committed by Linus Torvalds
Browse files

mm/page_alloc: optimize code layout for __alloc_pages_bulk 

Looking at perf-report and ASM-code for __alloc_pages_bulk() it is clear
that the code activated is suboptimal.  The compiler guesses wrong and
places unlikely code at the beginning.  Due to the use of WARN_ON_ONCE()
macro the UD2 asm instruction is added to the code, which confuse the
I-cache prefetcher in the CPU.

[mgorman@techsingularity.net: minor changes and rebasing]

Link: https://lkml.kernel.org/r/20210325114228.27719-5-mgorman@techsingularity.net


Signed-off-by: default avatarJesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: default avatarMel Gorman <mgorman@techsingularity.net>
Reviewed-by: default avatarAlexander Lobakin <alobakin@pm.me>
Acked-By: default avatarVlastimil Babka <vbabka@suse.cz>
Cc: Alexander Duyck <alexander.duyck@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Chuck Lever <chuck.lever@oracle.com>
Cc: David Miller <davem@davemloft.net>
Cc: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent 0f87d9d3

mm/page_alloc.c

+3 −3
Original line number Diff line number Diff line
@@ -5042,7 +5042,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, 
	unsigned int alloc_flags = ALLOC_WMARK_LOW;

	int nr_populated = 0;



	if (WARN_ON_ONCE(nr_pages <= 0))

	if (unlikely(nr_pages <= 0))

		return 0;



	/*
@@ -5089,7 +5089,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, 
	 * If there are no allowed local zones that meets the watermarks then

	 * try to allocate a single page and reclaim if necessary.

	 */

	if (!zone)

	if (unlikely(!zone))

		goto failed;



	/* Attempt the batch allocation */
@@ -5107,7 +5107,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, 


		page = __rmqueue_pcplist(zone, ac.migratetype, alloc_flags,

								pcp, pcp_list);

		if (!page) {

		if (unlikely(!page)) {

			/* Try and get at least one page */

			if (!nr_populated)

				goto failed_irq;