ionic: clean irq affinity on queue deinit

[ Upstream commit b9c17d39 ]

Add a little more cleanup when tearing down the queues.

Fixes: 1d062b7b ("ionic: Add basic adminq support")
Signed-off-by: Shannon Nelson <snelson@pensando.io>
Signed-off-by: David S. Miller <davem@davemloft.net>
Stable-dep-of: abfb2a58 ("ionic: remove WARN_ON to prevent panic_on_warn")
Signed-off-by: Sasha Levin <sashal@kernel.org>