Commit 8b43fd3d authored by Eric Dumazet's avatar Eric Dumazet Committed by Paolo Abeni
Browse files

net: optimize ____napi_schedule() to avoid extra NET_RX_SOFTIRQ 



____napi_schedule() adds a napi into current cpu softnet_data poll_list,
then raises NET_RX_SOFTIRQ to make sure net_rx_action() will process it.

Idea of this patch is to not raise NET_RX_SOFTIRQ when being called indirectly
from net_rx_action(), because we can process poll_list from this point,
without going to full softirq loop.

This needs a change in net_rx_action() to make sure we restart
its main loop if sd->poll_list was updated without NET_RX_SOFTIRQ
being raised.

Signed-off-by: default avatarEric Dumazet <edumazet@google.com>
Cc: Jason Xing <kernelxing@tencent.com>
Reviewed-by: default avatarJason Xing <kerneljasonxing@gmail.com>
Tested-by: default avatarJason Xing <kerneljasonxing@gmail.com>
Signed-off-by: default avatarPaolo Abeni <pabeni@redhat.com>
parent 821eba96

net/core/dev.c

+18 −4
Original line number Diff line number Diff line
@@ -4360,6 +4360,10 @@ static inline void ____napi_schedule(struct softnet_data *sd, 
	}



	list_add_tail(&napi->poll_list, &sd->poll_list);

	/* If not called from net_rx_action()

	 * we have to raise NET_RX_SOFTIRQ.

	 */

	if (!sd->in_net_rx_action)

		__raise_softirq_irqoff(NET_RX_SOFTIRQ);

}
@@ -6648,6 +6652,7 @@ static __latent_entropy void net_rx_action(struct softirq_action *h) 
	LIST_HEAD(list);

	LIST_HEAD(repoll);



start:

	sd->in_net_rx_action = true;

	local_irq_disable();

	list_splice_init(&sd->poll_list, &list);
@@ -6659,9 +6664,18 @@ static __latent_entropy void net_rx_action(struct softirq_action *h) 
		skb_defer_free_flush(sd);



		if (list_empty(&list)) {

			if (list_empty(&repoll)) {

				sd->in_net_rx_action = false;

			if (!sd_has_rps_ipi_waiting(sd) && list_empty(&repoll))

				barrier();

				/* We need to check if ____napi_schedule()

				 * had refilled poll_list while

				 * sd->in_net_rx_action was true.

				 */

				if (!list_empty(&sd->poll_list))

					goto start;

				if (!sd_has_rps_ipi_waiting(sd))

					goto end;

			}

			break;

		}