Commit 114518eb authored by Jens Axboe's avatar Jens Axboe Committed by Thomas Gleixner
Browse files

task_work: Use TIF_NOTIFY_SIGNAL if available 



If the arch supports TIF_NOTIFY_SIGNAL, then use that for TWA_SIGNAL as
it's more efficient than using the signal delivery method. This is
especially true on threaded applications, where ->sighand is shared across
threads, but it's also lighter weight on non-shared cases.

io_uring is a heavy consumer of TWA_SIGNAL based task_work. A test with
threads shows a nice improvement running an io_uring based echo server.

stock kernel:
0.01% <= 0.1 milliseconds
95.86% <= 0.2 milliseconds
98.27% <= 0.3 milliseconds
99.71% <= 0.4 milliseconds
100.00% <= 0.5 milliseconds
100.00% <= 0.6 milliseconds
100.00% <= 0.7 milliseconds
100.00% <= 0.8 milliseconds
100.00% <= 0.9 milliseconds
100.00% <= 1.0 milliseconds
100.00% <= 1.1 milliseconds
100.00% <= 2 milliseconds
100.00% <= 3 milliseconds
100.00% <= 3 milliseconds
1378930.00 requests per second
~1600% CPU

1.38M requests/second, and all 16 CPUs are maxed out.

patched kernel:
0.01% <= 0.1 milliseconds
98.24% <= 0.2 milliseconds
99.47% <= 0.3 milliseconds
99.99% <= 0.4 milliseconds
100.00% <= 0.5 milliseconds
100.00% <= 0.6 milliseconds
100.00% <= 0.7 milliseconds
100.00% <= 0.8 milliseconds
100.00% <= 0.9 milliseconds
100.00% <= 1.2 milliseconds
1666111.38 requests per second
~1450% CPU

1.67M requests/second, and we're no longer just hammering on the sighand
lock. The original reporter states:

"For 5.7.15 my benchmark achieves 1.6M qps and system cpu is at ~80%.
 for 5.7.16 or later it achieves only 1M qps and the system cpu is is
 at ~100%"

with the only difference there being that TWA_SIGNAL is used
unconditionally in 5.7.16, since it's required to be able to handle the
inability to run task_work if the application is waiting in the kernel
already on an event that needs task_work run to be satisfied. Also see
commit 0ba9c9ed.

Reported-by: default avatarRoman Gershman <romger@amazon.com>
Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
Signed-off-by: default avatarThomas Gleixner <tglx@linutronix.de>
Reviewed-by: default avatarOleg Nesterov <oleg@redhat.com>
Link: https://lore.kernel.org/r/20201026203230.386348-5-axboe@kernel.dk
parent 12db8b69

kernel/task_work.c

+29 −12
Original line number Diff line number Diff line
@@ -5,6 +5,34 @@ 


static struct callback_head work_exited; /* all we need is ->next == NULL */



/*

 * TWA_SIGNAL signaling - use TIF_NOTIFY_SIGNAL, if available, as it's faster

 * than TIF_SIGPENDING as there's no dependency on ->sighand. The latter is

 * shared for threads, and can cause contention on sighand->lock. Even for

 * the non-threaded case TIF_NOTIFY_SIGNAL is more efficient, as no locking

 * or IRQ disabling is involved for notification (or running) purposes.

 */

static void task_work_notify_signal(struct task_struct *task)

{

#if defined(TIF_NOTIFY_SIGNAL)

	set_notify_signal(task);

#else

	unsigned long flags;



	/*

	 * Only grab the sighand lock if we don't already have some

	 * task_work pending. This pairs with the smp_store_mb()

	 * in get_signal(), see comment there.

	 */

	if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) &&

	    lock_task_sighand(task, &flags)) {

		task->jobctl |= JOBCTL_TASK_WORK;

		signal_wake_up(task, 0);

		unlock_task_sighand(task, &flags);

	}

#endif

}



/**

 * task_work_add - ask the @task to execute @work->func()

 * @task: the task which should run the callback
@@ -28,7 +56,6 @@ int 
task_work_add(struct task_struct *task, struct callback_head *work, int notify)

{

	struct callback_head *head;

	unsigned long flags;



	do {

		head = READ_ONCE(task->task_works);
@@ -42,17 +69,7 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify) 
		set_notify_resume(task);

		break;

	case TWA_SIGNAL:

		/*

		 * Only grab the sighand lock if we don't already have some

		 * task_work pending. This pairs with the smp_store_mb()

		 * in get_signal(), see comment there.

		 */

		if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) &&

		    lock_task_sighand(task, &flags)) {

			task->jobctl |= JOBCTL_TASK_WORK;

			signal_wake_up(task, 0);

			unlock_task_sighand(task, &flags);

		}

		task_work_notify_signal(task);

		break;

	}