Commit 9b4bc85a authored by Vlastimil Babka's avatar Vlastimil Babka
Browse files

mm, slub: simplify kmem_cache_cpu and tid setup 



In slab_alloc_node() and do_slab_free() fastpaths we need to guarantee that
our kmem_cache_cpu pointer is from the same cpu as the tid value. Currently
that's done by reading the tid first using this_cpu_read(), then the
kmem_cache_cpu pointer and verifying we read the same tid using the pointer and
plain READ_ONCE().

This can be simplified to just fetching kmem_cache_cpu pointer and then reading
tid using the pointer. That guarantees they are from the same cpu. We don't
need to read the tid using this_cpu_read() because the value will be validated
by this_cpu_cmpxchg_double(), making sure we are on the correct cpu and the
freelist didn't change by anyone preempting us since reading the tid.

Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Acked-by: default avatarMel Gorman <mgorman@techsingularity.net>
parent 1572df7c

mm/slub.c

+9 −13
Original line number Diff line number Diff line
@@ -2882,15 +2882,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, 
	 * reading from one cpu area. That does not matter as long

	 * as we end up on the original cpu again when doing the cmpxchg.

	 *

	 * We should guarantee that tid and kmem_cache are retrieved on

	 * the same cpu. It could be different if CONFIG_PREEMPTION so we need

	 * to check if it is matched or not.

	 * We must guarantee that tid and kmem_cache_cpu are retrieved on the

	 * same cpu. We read first the kmem_cache_cpu pointer and use it to read

	 * the tid. If we are preempted and switched to another cpu between the

	 * two reads, it's OK as the two are still associated with the same cpu

	 * and cmpxchg later will validate the cpu.

	 */

	do {

		tid = this_cpu_read(s->cpu_slab->tid);

	c = raw_cpu_ptr(s->cpu_slab);

	} while (IS_ENABLED(CONFIG_PREEMPTION) &&

		 unlikely(tid != READ_ONCE(c->tid)));

	tid = READ_ONCE(c->tid);



	/*

	 * Irqless object alloc/free algorithm used here depends on sequence
@@ -3164,11 +3163,8 @@ static __always_inline void do_slab_free(struct kmem_cache *s, 
	 * data is retrieved via this pointer. If we are on the same cpu

	 * during the cmpxchg then the free will succeed.

	 */

	do {

		tid = this_cpu_read(s->cpu_slab->tid);

	c = raw_cpu_ptr(s->cpu_slab);

	} while (IS_ENABLED(CONFIG_PREEMPTION) &&

		 unlikely(tid != READ_ONCE(c->tid)));

	tid = READ_ONCE(c->tid);



	/* Same with comment on barrier() in slab_alloc_node() */

	barrier();