Commit 5c3b74a9 authored Jun 06, 2023 by Eric Dumazet Committed by David S. Miller Jun 07, 2023

rfs: annotate lockless accesses to RFS sock flow table



Add READ_ONCE()/WRITE_ONCE() on accesses to the sock flow table.

This also prevents a (smart ?) compiler to remove the condition in:

if (table->ents[index] != newval)
        table->ents[index] = newval;

We need the condition to avoid dirtying a shared cache line.

Fixes: fec5e652 ("rfs: Receive Flow Steering")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Simon Horman <simon.horman@corigine.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@amazon.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

parent 1e5c647c

include/linux/netdevice.h

+5 −2

Original line number	Diff line number	Diff line
		@@ -768,8 +768,11 @@ static inline void rps_record_sock_flow(struct rps_sock_flow_table *table,
		/* We only give a hint, preemption can change CPU under us */
		val \|= raw_smp_processor_id();

		if (table->ents[index] != val)
		table->ents[index] = val;
		/* The following WRITE_ONCE() is paired with the READ_ONCE()
		* here, and another one in get_rps_cpu().
		*/
		if (READ_ONCE(table->ents[index]) != val)
		WRITE_ONCE(table->ents[index], val);
		}
		}

net/core/dev.c

+4 −2

Original line number	Diff line number	Diff line
		@@ -4471,8 +4471,10 @@ static int get_rps_cpu(struct net_device dev, struct sk_buff skb,
		u32 next_cpu;
		u32 ident;

		/* First check into global flow table if there is a match */
		ident = sock_flow_table->ents[hash & sock_flow_table->mask];
		/* First check into global flow table if there is a match.
		* This READ_ONCE() pairs with WRITE_ONCE() from rps_record_sock_flow().
		*/
		ident = READ_ONCE(sock_flow_table->ents[hash & sock_flow_table->mask]);
		if ((ident ^ hash) & ~rps_cpu_mask)
		goto try_rps;