revert: "softirq: Let ksoftirqd do its job"

Due to the mentioned commit, when the ksoftirqd processes take charge
of softirq processing, the system can experience high latencies.

In the past a few workarounds have been implemented for specific
side-effects of the above:

commit 1ff68820 ("watchdog: core: make sure the watchdog_worker is not deferred")
commit 8d5755b3 ("watchdog: softdog: fire watchdog even if softirqs do not get to run")
commit 217f6974 ("net: busy-poll: allow preemption in sk_busy_loop()")
commit 3c53776e ("Mark HI and TASKLET softirq synchronous")

but the latency problem still exists in real-life workloads, see the
link below.

The reverted commit intended to solve a live-lock scenario that can now
be addressed with the NAPI threaded mode, introduced with commit
29863d41 ("net: implement threaded-able napi poll loop support"),
and nowadays in a pretty stable status.

While a complete solution to put softirq processing under nice resource
control would be preferable, that has proven to be a very hard task. In
the short term, remove the main pain point, and also simplify a bit the
current softirq implementation.

Note that this change also reverts commit 3c53776e ("Mark HI and
TASKLET softirq synchronous") and commit 1342d808

 ("softirq: Don't
skip softirq execution when softirq thread is parking"), which are
direct follow-ups of the feature commit. A single change is preferred to
avoid known bad intermediate states introduced by a patch series
reverting them individually.

Link: https://lore.kernel.org/netdev/305d7742212cbe98621b16be782b0562f1012cb6.camel@redhat.com/
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
Tested-by: Jason Xing <kerneljasonxing@gmail.com>
Reviewed-by: Jakub Kicinski <kuba@kernel.org>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Link: https://lore.kernel.org/r/57e66b364f1b6f09c9bc0316742c3b14f4ce83bd.1683526542.git.pabeni@redhat.com
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
(cherry picked from commit b8a04a538ed4755dc97c403ee3b8dd882955c98c)
Signed-off-by: Clark Williams <clark.williams@gmail.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>