Commit 8d970396 authored Nov 23, 2021 by Frederic Weisbecker Committed by Paul E. McKenney Dec 09, 2021

rcu/nocb: Prepare nocb_cb_wait() to start with a non-offloaded rdp



In order to be able to toggle the offloaded state from cpusets, a nocb
kthread will need to be created for all possible CPUs whenever either
of the "rcu_nocbs=" or "nohz_full=" parameters are specified.

Therefore, the nocb_cb_wait() kthread must be prepared to start running
on a de-offloaded rdp.  To accomplish this, simply move the sleeping
condition to the beginning of the nocb_cb_wait() function, which prevents
this kthread from attempting to invoke callbacks before the corresponding
CPU is offloaded.

Reviewed-by: Neeraj Upadhyay <quic_neeraju@quicinc.com>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Uladzislau Rezki <urezki@gmail.com>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Joel Fernandes <joel@joelfernandes.org>
Tested-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

parent 2ebc45c4

kernel/rcu/tree_nocb.h

+12 −11

Original line number	Diff line number	Diff line
		@@ -803,6 +803,18 @@ static void nocb_cb_wait(struct rcu_data *rdp)
		bool can_sleep = true;
		struct rcu_node *rnp = rdp->mynode;

		do {
		swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
		nocb_cb_wait_cond(rdp));

		// VVV Ensure CB invocation follows _sleep test.
		if (smp_load_acquire(&rdp->nocb_cb_sleep)) { // ^^^
		WARN_ON(signal_pending(current));
		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty"));
		}
		} while (!nocb_cb_can_run(rdp));


		local_irq_save(flags);
		rcu_momentary_dyntick_idle();
		local_irq_restore(flags);
		@@ -855,17 +867,6 @@ static void nocb_cb_wait(struct rcu_data *rdp)

		if (needwake_state)
		swake_up_one(&rdp->nocb_state_wq);

		do {
		swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
		nocb_cb_wait_cond(rdp));

		// VVV Ensure CB invocation follows _sleep test.
		if (smp_load_acquire(&rdp->nocb_cb_sleep)) { // ^^^
		WARN_ON(signal_pending(current));
		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty"));
		}
		} while (!nocb_cb_can_run(rdp));
		}

		/*