spi: Only defer to thread for cleanup when needed

Currently we always defer idling of controllers to the SPI thread, the goal
being to ensure that we're doing teardown that's not suitable for atomic
context in an appropriate context and to try to batch up more expensive
teardown operations when the system is under higher load, allowing more
work to be started before the SPI thread is scheduled. However when the
controller does not require any substantial work to idle there is no need
to do this, we can instead save the context switch and immediately mark
the controller as idle. This is particularly useful for systems where there
is frequent but not constant activity.

Signed-off-by: Mark Brown <broonie@kernel.org>
Link: https://lore.kernel.org/r/20200715163610.9475-1-broonie@kernel.org


Signed-off-by: Mark Brown <broonie@kernel.org>