Commit 975707f2 authored by Peter Zijlstra's avatar Peter Zijlstra
Browse files

sched: Prepare to use balance_push in ttwu() 



In preparation of using the balance_push state in ttwu() we need it to
provide a reliable and consistent state.

The immediate problem is that rq->balance_callback gets cleared every
schedule() and then re-set in the balance_push_callback() itself. This
is not a reliable signal, so add a variable that stays set during the
entire time.

Also move setting it before the synchronize_rcu() in
sched_cpu_deactivate(), such that we get guaranteed visibility to
ttwu(), which is a preempt-disable region.

Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: default avatarValentin Schneider <valentin.schneider@arm.com>
Tested-by: default avatarValentin Schneider <valentin.schneider@arm.com>
Link: https://lkml.kernel.org/r/20210121103506.966069627@infradead.org
parent 640f17c8

kernel/sched/core.c

+6 −5
Original line number Diff line number Diff line
@@ -7320,6 +7320,7 @@ static void balance_push_set(int cpu, bool on) 
	struct rq_flags rf;



	rq_lock_irqsave(rq, &rf);

	rq->balance_push = on;

	if (on) {

		WARN_ON_ONCE(rq->balance_callback);

		rq->balance_callback = &balance_push_callback;
@@ -7489,17 +7490,17 @@ int sched_cpu_deactivate(unsigned int cpu) 
	int ret;



	set_cpu_active(cpu, false);

	balance_push_set(cpu, true);



	/*

	 * We've cleared cpu_active_mask, wait for all preempt-disabled and RCU

	 * users of this state to go away such that all new such users will

	 * observe it.

	 * We've cleared cpu_active_mask / set balance_push, wait for all

	 * preempt-disabled and RCU users of this state to go away such that

	 * all new such users will observe it.

	 *

	 * Do sync before park smpboot threads to take care the rcu boost case.

	 */

	synchronize_rcu();



	balance_push_set(cpu, true);



	rq_lock_irqsave(rq, &rf);

	if (rq->rd) {

		update_rq_clock(rq);

kernel/sched/sched.h

+1 −0
Original line number Diff line number Diff line
@@ -975,6 +975,7 @@ struct rq { 
	unsigned long		cpu_capacity_orig;



	struct callback_head	*balance_callback;

	unsigned char		balance_push;



	unsigned char		nohz_idle_balance;

	unsigned char		idle_balance;